N. Henze and W. Nejdl
University of Hannover, Lange Laube 3
30159 Hannover, Germany
phone: 0049-511-762 9711, fax: 0049-511-7629712,
{henze, nejdl}@kbs.uni-hannover.de
 
 
 

Internet-based teaching and learning has been the topic of many recent projects and research work. While it is relatively easy to use the Internet to transport conventional types of lectures and course notes, it is more difficult to utilize the full power of Internet-based techniques to advance teaching and learning conceptually. In this paper we will discuss the KBS Virtual Learning Environment built within our group, its theoretical background in the form of constructivist models of learning and teaching, its use for our courses and its implementation.
 
 

Distributed Learning Environments
 
 
 

1 INTRODUCTION

Within the KBS Virtual Classroom Project, we have been working on virtual learning environments based on the Internet and the World Wide Web since 1996. One main goal for this project has always been to utilize the full power of these techniques to innovat e teaching and learning for our courses, instead of just transplanting ordinary lectures onto the Internet. In this paper we will discuss the educational issues motivating our approach and the concepts and technical issues to implement such a learning envi ronment.

We start by discussing our approach of goal-based scenarios within this framework. We analyze the concepts and properties necessary to use goal-based scenarios in our teaching environment and how we apply them for our courses and exercises. We will disc uss the techniques to support such an environment and how we implement them on top of the Internet and World Wide Web. We conclude with a preliminary evaluation of recent courses.
 
 

2 Constructivistic Models of Learning

Computational learning environments benefit from a strong background from educational theory. Simply reproducing conventional teaching and learning concepts in a computational environment does not utilize these new technologies. Educational models, which s how particularly interesting features for many parts of academic education are constructivist models of learning and teaching (see also the discussion of the relationship between such models and the field of instructional technology in (Duffy, 1992)).

Critical elements in the design of constructivist learning environments are the specification and use of authentic and complex activities during the learning process (Fishman, 1991). Authentic activities shift part of the ownership for learning and perf ormance from the teacher to the student. While working on their tasks in real-world situations, students have to learn and apply the required skills at least partly by themselves. This does not allow learners to passively consume a lecture without critical reflection, learned knowledge and skills are needed to perform the task which stimulates the students to think about the arising problems and techniques. The teacher's role is to coach the students and to describe the initial concepts necessary for their task.

Real-world examples are characterized by the complex context in which they arise: few applications of techniques and concepts occur in the simplified forms used for highly abstracted exercise problems often presented to learners (see also the discussion in (Duffy, 1992). While abstraction is of course necessary and small exercises can be used to discuss specific issues, project-based learning has to be used to rebuild real-world complexity. The global project context determines the learners perspective o n a given task, while subtasks in a smaller context provide guidance of the learning process.

The ability to develop multiple and alternative perspectives on a problem is also a central skill for performing tasks. Collaborative learning promotes the exchange and reflection on different views. As project work is often done in teams, learners trai n their capabilities for team-work and collaboration.
 
 

2.1 Project-Based Learning Approaches

Constructive learning models and project-based learning can be supported by different approaches and can be viewed from different perspectives. We will sketch some of these starting from the concepts discussed in (Schank, 1994).

Simulation-Based Learning By Doing: Acquisition of knowledge is guided by goals or projects actively pursued by the students. Knowledge and techniques are learned and used to fulfil specific tasks which are needed to reach the project goals. Teachers ha ve to give help when needed. To support learning by doing, simulations of all kinds of tasks can be built.

Incidental Learning: Projects and goals of a course have to be selected by the teacher in such a way that skills and knowledge needed to pursue these goals correspond to the (conventional) course content. Obviously, in a task oriented learning environme nt not only a single set of techniques leads to project success. So, although the base set of knowledge and skills that will be learnt is set, students can individually control their learning, depending on their previous knowledge and their individual pref erences.

Learning by Reflection: Students are encouraged to reflect on given problems and on different solutions found by themselves or other student groups. Continuous discussion of teachers and students leads to original solutions and new insights.

Case-Based Teaching: Presentation of knowledge by the teaching staff depends to a certain extent on the progress students make in solving the given problems. Support is oriented mainly around cases with attached related knowledge, facts and problem solv ing methods. These cases can be continuously added to the knowledge base of the learning environment and represent an increasing knowledge and support base for the students.

Learning by Exploring: Communication between teachers and students is a very important aspect in this approach. Topics of the course are discussed in study groups or with the teacher. Small learning units contain the basic knowledge needed. Learning uni ts are presented in the knowledge base which is extended by students and teachers. Students are engaged to study and to find out facts, skills, and research results on their own.
 
 

2.2 Goal-Based Scenarios

To pursue the different approaches discussed in the previous section, we use the idea of goal-based scenarios, also discussed in (Schank, 1994). Goal-based scenarios are used as a framework for our learning environment and provide both the scenario context which models real-world applications as well as the scenario structure which settles the features of an adaptive learning environment.

Within the goal-based scenario framework we provide the context, motivation and material needed to work on specific projects. These projects are chosen such that a basic set of knowledge and techniques is needed to solve them plus additional in-depth kn owledge about specific issues related to this basic set. Tutorial help is given by the teachers which answer the questions of their students, if possible at the moment the questions are arising (just in time). Students are encouraged to discuss approaches and solutions in small working groups. Additionally, students have access to a knowledge base about the subject. This knowledge base contains studying material, ideally centered around cases, complete with solutions, additional information and knowledge an d problem solving strategies. Both teachers and students extend the knowledge base, which therefore also contains a repository of former projects. This serves to connect new problems and already solved cases and provides easy access to needed materials.

The role of teachers as coaches and mentors implies that students have access to teachers besides fixed lessons during the week. Similarly a knowledge base will not be effective, if access to it is restricted to a few hours during the week. Team-work on real-world problems cannot be interrupted by missing communication. All these points are critical to constructivistic learning environments but can usually not be sufficiently realized in a traditional environment.

To support such a goal-based scenario approach, Internet and World Wide Web technologies as well as various database, artificial intelligence and hypermedia techniques allow us to build learning environments offering the following facilities:

Availability of working environment and communication facilities.
Students and teachers have anytime-access to the working environment which is needed to solve their tasks. Communication and discussion facilities also have to be continuously available for discussing course related topics.

Team-oriented learning.
Small learning groups working on specific projects allow collaborative learning, increasing learning efficiency and team working capabilities.

Personal mentoring.
Students need to have a mentor that can supervise and help them individually during their projects. Discussion within working groups is similarly encouraged. The learning environment has to enable flexible and time-independent communication between mentors and their students.

Distributed time-independent access to the course knowledge base.
Students should have anytime-access to the knowledge base, both for reference as well as for extension.

Adaptive course knowledge bases.
Students have to use the knowledge base in different ways, depending on project progress and previous knowledge. The knowledge base therefore should have adaptive capability, offering additional help for beginners and in-depth knowledge for experts, as wel l as different routes and indices for different purposes and readers.

Extensible course knowledge bases.
As the knowledge base can be extended both by teachers and students, its structure, navigation paths, visualization and user adaptivity has to be modeled explicitly to allow easy extension by simple instantiation of models (similar in purpose to database s chemas).
 
 

3 Our Current Virtual Learning Environment

In this section, we will discuss the main features of our current learning environment, following the requirements introduced in the previous section. The learning environment is being continuously enhanced for all our courses and laboratories. We will sta rt with features available for all our courses, and then describe specific course instances.
 
 

3.1 Availability of the Working Environment

In order to make the working environment continuously available, access to all parts of the working environment (in this case background and project information as well as the necessary software tools) are available over the Internet. Access to the Interne t for students is either from campus computers, over phone lines and terminal servers (administrated by the university computing center and the students themselves), and by way of Internet providers.

All necessary tools are available for most current operating systems, so students can use them locally at every computer they have access to, including of course their home computers. If license restrictions make this impossible (as in the case of a lar ge software engineering tool), at least anytime access over the internet is available (currently with X11 interface). Locally used tools are always internet-based, so access to central servers, repository and communication facilities is always possible.
 
 

3.2 Project-Based Learning

Most current courses and laboratories have specific large-scale projects, tailored specifically to the course contents. The current programming laboratory for undergraduates for example consists of two programming projects within one semester, the software engineering courses focuses on one larger project spanning two semesters (using the development of an e-mail client to practically illustrate the whole software process), an second course in artificial intelligence and an advanced programming laboratory f ocus on intelligent agents (architectures, possibilities, and development of specific agents).

All project results are presented on the World Wide Web, where they are available for other groups as well as for students in future semesters. This motivates students to not only solve larger real-world problems, but also elaborate them in a way suitab le for external presentation. Results for the various parts of the projects are usually published as soon as they are available, to encourage discussion of intermediate results between student groups.
 
 

3.3 Team-Oriented Learning and Mentoring

Project-based learning is done in groups of two to four students. Groups are formed at the beginning of each semester. Students in such a group work on a common project and present their results together. Collaborative learning and working is encouraged. F or each group we assign a personal mentor, which can either be a graduate student, a Ph.D. student or the professor. This mentor is available personally at specific hours during the weeks, or anytime by electronic communication facilities. Group meetings a nd discussions are possible in person, or electronically. Documents can be managed on a central repository with version management facilities (with distribution and update to local computers managed over the Internet) as well as on our WWW server.
 
 

3.4 Electronic Communication Facilities

Each student group has a group communication center, which includes e-mail lists to all students within this group (plus another one including the group mentor), a communication room on the WWW (read and write access restricted to members of the group), a presentation room on the WWW (which is readable by everyone), and a central repository (based on a client/server version of CVS) with configuration and version management.

Each course includes three discussion groups (implemented as newsgroups), one for official announcements (Announcements), one for general discussion and questions as well as course oriented exercises plus student answers (Discussion Forum), and one for free communication not directly related to the course ( Cyber Cafe). The Announcements and Discussion Forum are also automatically archived, indexed and made available over the WWW. Synchronous communication at present can take place over a text based chat tool, the KBS Online Chat Forum.

All facilities / client programs are available on all supported operating systems and computers connected to the Internet.
 
 

4 Network Environment

Figure 1 shows the hardware/networking environment underlying our solution. The central server located at our institute serves as the central repository for all course-related material like lecture slides, tutorials, web pages, hyperbook, programs etc. All central repositories (CVS, WWW communication and presentation area etc.) are also stored at the institute's server. A variety of working environments for accessing the data are supported: Classically working on a client at the institute and accessing the data through Ethernet/NFS; accessing data from home PC's via a modem connection to a dedicated machine in the institute's network; accessing data from the university computer pool or from the (student's) home PC connecting via modem to a dedicated student server (configured to support a large number of parallel modem and ISDN connections).
 

5 Adaptive Hyperbook

In this section we present the initial design of a hyperbook for the courses Introduction to Computer Programming and Software Engineering. In contrast to the rest of this paper, the hyperbook system described in this chapter is not yet in production use, but is currently being written and implemented. We hope to be able to use it starting next semester. The modeling language and methodology as well as the underlying system is described in more detail in (Fröhlich, 1997a, 1997b, 1997c). In this section, we will discuss the general principles and the use of the hyperbook within our learning environment.

The goal of the hyperbook system is to provide an adaptive extendable environment / hyperbook, centered around projects and cases, which tightly integrates case descriptions and solutions, concept descriptions and more conventional tutorial notes etc. a nd contains all material needed for the courses and the projects. As the hyperbook has to be extensible by different authors as well as by students, a systematic model of all aspects of the hyperbook (domain structure, navigation, visualization and adaptat ion) is necessary. All these details are declaratively defined in the various models (including navigation through links, indices etc).

In the last years, several methodologies for hypermedia modeling have been developed, among which the Hypertext Design Method (HDM) (Garzotto, 1993), the Relationship Management Methodology (RMM) (Isakowitz, 1995), and the Object-Oriented Hypermedia Des ign Model (OOHDM) (Schwabe, 1996) are the most prominent examples. These methods provide primitives for modeling a hypermedia application domain. Based on this domain model the possibilities for navigation are described. As pointed out in (Isakowitz, 1995) , such methods are best suited for designing front ends to loosely structured data. The navigational concepts proposed in these methods are geared towards indexing a large amount of relatively simple information pieces. On the other hand, educational hyper books have to include various forms of (sequential) tours, user models to adapt to the user's experience and visualization models.
 

On the other hand, current work in adaptive hypermedia textbooks (such as (Kay, 1994), (Brusilovsky, 1994a, 1996a, 1996b)) concentrates on building domain models and user models for adaptive hyperbooks, using semantic nets to describe these domain models a nd to index the hyperbook with the corresponding domain nodes. Our work builds on these concepts and focuses on the modeling language and meta-language for building domain models, explicit navigation and visualization models and user models.

For the declarative representation of the hyperbook data models we use a dialect of the object-oriented conceptual modeling language Telos (Mylopoulos, 1990), which is implemented in the ConceptBase system (Jarke, 1995). This language combines object-or iented concepts with deductive rules and constraints. Due to its representational power Telos is suitable for meta modeling, i.e. for describing domain-specific modeling languages (Nissen, 1996). In this spirit, we have used a Telos meta model to define th e primitives used for domain modeling, navigational modeling, user modeling and visualization modeling, as shown in figure 2.

The domain model describes the concepts of the application domain, in our case centered around the setting of a task. A small part of the domain model of the software engineering section is shown in figure 3. It shows the definition of a Software Proces s consisting of several Phases, (1:n relationship between Software Process and Phase). This relationship is indexed by the key attribute PhaseNr (Phase Number). This key attribute specifies the order in which the Phases are presented. Each Phase in turn ha s several Activities. The bottom part of the figure shows specific instantiations for the OMT software process.
 

The relationships defined in the domain model are the basis for systematic semantic-based navigation in the hyperbook. The navigational model specifies the details of navigating the book, using a set of predefined concepts. All hyperlinks in the hyperbook document are generated automatically from the description of the hyperbook in the database.

The visualization model is built upon WWW pages, page fragments and mime objects, and defines how the material in the hyperbook is presented. The user model is based on a description of prerequisite knowledge (describing the relationship between initial and more advanced concepts) and a simple overlay model like the one used in (Brusilovsky, 1994b). Therefore the presentation can adapt to different knowledge levels of the learner, as well as different preferences.

Figure 4 shows the basic system architecture, which is based on standard components complemented by a server-side applet and the ConceptBase database manager. After logging into the system by activating the login applet a user-specific starting page is displayed. The user navigates the hyperbook by activating links. Whenever a hyperlink is activated the name of the corresponding domain object plus the name of the user are passed to a server-side applet. The applet queries the data base for the fragments of the page representing the domain object and for the domain object's navigational possibilities. From this information it constructs a user specific page. It does so by combining the visible fragments in a HTML page. Then it updates the user model by mar king this page as read and uploads the page to the client, where it is displayed in an ordinary WWW browser.
 

6 Specific Courses
 

6.1 Introduction to Computer Programming

This introductory course for undergraduates particularly emphasized the working environment (complete working environment available locally for each student computer as well as for university computers), communication and discussion over the Internet (with weekly questions/exercises and student answers in the Discussion Forum), and group work (four students in each group). Projects in this course were quite small, but will be larger in the next semester.

The accompanying programming laboratory during the summer semester focused on larger projects (two during the semester, with five alternative projects), and was run completely over the Internet (including electronic group communication and presentation facilities), with personal student mentors for each group (and personal meetings of groups and mentors as well). All project information as well as additional information about Ada 95 (taught during the winter semester and used for the first project) and J ava and C++ (learned during the programming lab and used for the second project) was available online.

The screenshot (figure 5) shows an example of the work with the virtual learning environment in this course: The course's homepage can be seen in the middle, a programming tutorial (public domain) with a quiz belonging to each chapter (on the left hand) , a part of the newsgroup Discussion Forum (upper right side) and a question posed to the mentor (lower right side).
 
 

6.2 Introduction to Software Engineering

This set of courses covers two semesters, and includes lecture and exercise hours. Presentation of course contents is focused on the basic knowledge needed for a software engineering project, and is experienced within a specific project context, namely dev eloping an e-mail client. The first semester focuses on analysis and design issues, the second on implementation, testing and some more advanced issues. The project is done in student groups of three to four people, with a Ph.D. student and the professor a s personal mentors. Discussion of solution possibilities, alternatives and results is done as needed, presentation of results in the WWW is done weekly, the semester finishes with an evaluation of the different projects by the student groups.
 
 

6.3 Introduction to Artificial Intelligence

This set of courses also covers two semesters, the first one centered around general AI techniques, the current one and an graduate laboratory focus on the in-depth analysis of intelligent agents in form of a student projects (from the theoretical analysis of agents, agent architectures and frameworks to specific implementations). Again, students work in groups of two to three, with a personal mentor and regular discussion of results. While the last AI course (taught for the first time and only for a few st udents) was hampered by a very inhomogeneous student group, the project results within the lab were very encouraging and will be used in further courses.
 

7 Evaluation

So far, we have formally evaluated part of our environment within the course Introduction to Computer Science and Programming, which stretches over two semesters. This course is taken by about 100 students mainly studying electrical engineering and technic al computer science. The first semester includes a two-hour lecture and one hour of exercises, the second semester includes another two hours of exercises in the form of a programming laboratory.

In winter 96/97 we supported a course and exercises with our virtual learning environment. A questionnaire (Henze, 1997) in this semester showed that our concept was mostly accepted by the students and improved the student's learning progress.

One part of the questionnaire was concerned with the structure of the groups, their working processes and communication flow. The investigation showed the different knowledge of the students at the beginning of the course: More than 40% of the students already had good or very good programming skills, about 30% said they had only little experience while the others had no programming experience at all. All but one of the groups consisted of students with different programming skills.
   Students in the course liked the possibility to reach their mentor anytime during the week and contacted them very frequently. About 20% of the groups used only email to contact their mentor while 50% of the groups wanted additional personal meetings wi th their mentor. However, communication between the members of the group themselves was done in different ways: About 70% of the groups did their exercises mainly on the basis of personal meetings, no group exclusively used email, 10% worked by phone.
   Another part of the questionnaire was aimed at the comparison of the groups, their performance and interest. The mentor's opinion about their groups performance together with the results the group members reached in the examinations during the course sh owed that about 10% of the groups could be described as very good, 45% as good, 35% as average and 10% as only poorly interested. The attempt to characterize good groups led to only one significant difference in comparison to the other groups belonging to the course: Good groups are good at teamwork. This further shows the necessity of encouraging collaborative working and project-based work among the students.

In summer 97 we supported the subsequent programming laboratory within the Virtual Learning Environment. Students who took part in the programming laboratory were involved in two large programming projects which each covered a half semester. It was the students responsibility to divide and delegate the project into single tasks among their group. 59 % of the students told that they had no problems to manage their projects, 23 % had more or less difficulties in organizing their common work. In a few case s collaboration within the group was very difficult as team members stopped working and left the group completely. We are currently discussing several alternatives for helping students work better within groups, as we see this as an important testing groun d for their further professional work after graduation.
   On the other hand about 71 % of the students told about good and very good teamwork. During the whole programming laboratory students usually used email to contact their mentors, only a quarter of the student groups had regular meetings besides email di scussions (which contrasted to their behavior during the programming lecture in the preceding semester).
   The version control system CVS (included in the summer semester) was not used as much as aspected. Two main reasons may explain this behavior: First, CVS was not introduced during the winter lecture, so students already installed their own ways to contr ol the program versions, e.g. by using email for sending the current version to other group members or by using a dedicated location in one user's directory with free access for the group. Second, although the projects covered half a semester they were pro bably still too small to really need advanced configuration and version management.
   A few groups documented the project progress among their group using the Communication Room. The final presentation of the projects in the group's Presentation Room (public access) was very successful: About 88 % of the students liked the possibility to present their work on the World Wide Web (the results can be seen at http://www.kbs.uni-hannover.de/ praktikum/ praktikum97/ teilnehmer.html). The projects presented there are fully documented: a (readable) implementation together with a complete documentat ion (README, INSTALL, description of the program structure and of used algorithms, a short user's manual) as well as test protocols can be found there. While we had introduced the basic structure (documentation, sources, etc.) the student groups showed gre at enthusiasm in presenting their work and their group.
 
 

8 Conclusion and Future Work

In this paper we have discussed the KBS Virtual Learning environment developed at our institute. We have analyzed the requirements for the design of this environment, resulting from constructivistic models of teaching and learning, and their implementation within our Internet-based Virtual Learning Environment has been shown. We also discussed the use of our environment and a preliminary evaluation of its use for our courses.

Future work will mainly concentrate on improving the overall concepts, and on continuing the design and implementation of our hyperbook environment to build really adaptive and extensible course knowledge bases. Especially within the area of hyperbook d esign a lot of research questions are still open and need to be addressed.
 
 

9 REFERENCES

Brusilovsky, P. and Pesin, L.(1994a) ISIS-Tutor: An Intelligent Learning  Environment for CDS/ISIS Users. In Proc. of CLCE'94, Joensuu, Finland.

Brusilovsky, P. (1994b) Adaptive hypermedia: an attempt to analise and generalize. In Workshop on Adaptive hypertext and hypermedia at UM'94, Hyannis, Cape Cod, MA, USA.

Brusilovsky, P., Schwarz, E. and Weber, G. (1996a) ELM-ART: An intelligent tutoring system on world wide web. In C. Frasson, G. Gauthier, and A. Lesgold,  editors, Intelligent Tutoring Systems (Lecture Notes in Computer Science, Vol. 1086), pages 261-269, Berlin, Springer.

Brusilovsky, P., Schwarz, E. and Weber, G.(1996b) A Tool for Developing Adaptive Electronic Textbooks on WWW. In Proceedings of WebNet'96 - World Conference of the Web Society, Boston, MA, USA.

Duffy,T. and Jonassen, D. (1992) Constructivism and the Technology of  Instruction. Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Fröhlich, P., Henze, N. and Nejdl, W. (1997a). Conceptual modeling for educational hyperbooks. In The 4th International Conference on Multimdedia Modelling, November 17 - 20, Singapore.

Fröhlich, P., Henze, N. and Nejdl, W. (1997b) Meta-modeling for hypermedia design. In Proc. of Second IEEE Metadata Conference, Maryland.

Fröhlich, P. and Nejdl, W.(1997c). A Database-Oriented Approach to the Design of Educational Hyperbooks. In Proceedings of the AIED'97 Workshop on Intelligent Educational Systems on the World Wide Web, Kobe, Japan.

Garzotto, F., Schwabe, D. and Paolini, P. (1993) HDM - A Model Based Approach to Hypermedia Application Design. ACM Transactions on Information Systems, 11(1).

Henze, N. (1997) Auswertung der Befragung der Teilnehmer der Vorlesung  Grundzüge der Informatik, Technical Report, University of Hannover.

Isakowitz, T., Stohr, E.A. and Balasubrahmanian, P. (1995) RMM: A methodology for structured hypermedia design. Communications of the ACM, 38(8).

Jarke, M., Gallersdörfer, R., Jeusfeld, M., Staudt, M. and Eherer,S. (1995) Conceptbase - a deductive object base for meta data anagement. Journal on Intelligent Information Systems, 4(2):167 - 192.

Kay, J. and Kummerfeld, R.J. (1994) An individualised course for the C programming language. In Proc. of the 2nd International World Wide Web Conference, Chicago, USA.

Mylopoulos, J., Borgida, A., Jarke, M. and Koubarakis, M. (1990) Telos: A language for representing knowledge about information systems. ACM Transactions on Information Systems, 8(4).

Nissen, H.W., Jeusfeld, M.A., Jarke, M., Zemanek, G.V., and Huber, H. (1996) Requirements analysis from multiple perspectives: Experiences with conceptual modeling technology. IEEE Software, 13(2).

Fishman, B.J., Honebein, P.C., Duffy, T.M. (1991) Constructivism and the design of learning enivronments: Context and authentic activities for learning. In NATO Advanced Workshop on the Design of Constructivist Learning Environments.

Schank, R. and Cleary, C. (1994) Engines for Education. Lawrence Erlbaum Associates, Hillsdale, New Jersey.

Schwabe, D., Rossi, G. and Barbosa, S.D.J. (1996) Systematic hypermedia application design with OOHDM. In Proceedings of the Seventh ACM Conference on Hypertext, Washington DC.
 
 

10 BIOGRAPHY
 
Dipl.-Math. Nicola Henze

Dipl.-Math. N. Henze has received her masters degree in mathematics from the University of Hannover. As a member of the knowledge-based systems group at the University of Hannover, she is working on adaptable and adaptive hypertext systems and user-adapted learning environments.
 

Prof. Dr. techn. Wolfgang Nejdl

Prof. Dr. Nejdl has graduated from the Technical University of Vienna in computer science, and is currently full professor for computer science at the University of Hannover, and head of the knowledge-based systems group. He has worked in the areas of data bases, artificial intelligence and hypermedia for education, and has published more than 90 articles in conference proceedings and journals on these subjects.