Advanced logging

By default, MISCore logs the events and durations from all built-in log_event() or log_duration() method in any Process in your model. However, it is also possible to include additional log statements whenever you modify or create a new Process yourself. This tutorial will briefly go over how to add log statements and how to log at the individual level.

In the example below, we constructed a new disease Process. It first draws a normally distributed age of disease onset. Next, the disease transitions to a clinical phase after 10 years. The person will die of the disease 5 years after that. Writing the script to run this Disease process and check the logged events and durations is left as an exercise to the reader.

 1from miscore import Process
 2
 3
 4class Disease(Process):
 5
 6    def __init__(self, onset_age_mean, onset_age_sd, onset_to_clinical=10,
 7                 clinical_to_death=5, name="disease"):
 8        # Store the parameters
 9        self.onset_age_mean = onset_age_mean
10        self.onset_age_sd = onset_age_sd
11        self.onset_to_clinical = onset_to_clinical
12        self.clinical_to_death = clinical_to_death
13        self.name = name
14
15        # Define how the process responds to messages
16        self.callbacks = {
17            "__start__": self.schedule_onset,
18            "disease_onset": self.onset,
19            "disease_clinical": self.clinical,
20        }
21
22    def properties(self, rng, n, properties):
23        # Draw the age of onset for each individual
24        onset_age = rng.normal(self.onset_age_mean, self.onset_age_sd, size=n)
25        onset_age[onset_age < 0] = 0
26
27        # Return the calculated properties
28        return {
29            "disease_onset_age": onset_age,
30        }
31
32    def schedule_onset(self, individual):
33        # Schedule onset event
34        individual.add_to_queue_age(
35            age=individual.properties("disease_onset_age"),
36            message="disease_onset"
37        )
38
39    def onset(self, individual):
40        # Schedule transition to clinical state
41        individual.add_to_queue(
42            delta=self.onset_to_clinical,
43            message="disease_clinical"
44        )
45
46    def clinical(self, individual):
47        # Save the current age in the memory
48        individual.memory["disease_clinical_age"] = individual.age
49
50        # Schedule death event
51        individual.add_to_queue(
52            delta=self.clinical_to_death,
53            message="disease_death",
54            terminate=True
55        )

Within a process, the log_event() and log_duration() methods can be used to log events and durations, respectively. When new tags are added, you have to include them in the event_tags and duration_tags lists. These should contain all tags that can be logged by a process.

Log events

In most cases, events have to be logged through the log_event() function. For example, let’s log the event disease_onset, which will tell us in the model output when the disease started. Moreover, we log the event disease_clinical, indicating when the disease became clinical. We should add these tags to the event_tags in the process and add a call to the log_event() function in the process when the event takes place.

The only exceptions which do not need to be explicitly logged by the log_event() function are terminal events. Terminal events are automatically logged if the tag is added to the event_tags of the Process. For example, the disease_death event, which is added to the event queue in line 63, is a terminal event because terminate is set to True.

As such, this Disease process will output 3 tags in the events DataFrame: disease_onset, disease_clinical and disease_death.

 1from miscore import Process
 2
 3
 4class Disease(Process):
 5    # All event tags that can be logged by this process
 6    event_tags = [
 7        "disease_onset",
 8        "disease_clinical",
 9        "disease_death",
10    ]
11
12    def __init__(self, onset_age_mean, onset_age_sd, onset_to_clinical=10,
13                 clinical_to_death=5, name="disease"):
14        # Store the parameters
15        self.onset_age_mean = onset_age_mean
16        self.onset_age_sd = onset_age_sd
17        self.onset_to_clinical = onset_to_clinical
18        self.clinical_to_death = clinical_to_death
19        self.name = name
20
21        # Define how the process responds to messages
22        self.callbacks = {
23            "__start__": self.schedule_onset,
24            "disease_onset": self.onset,
25            "disease_clinical": self.clinical,
26        }
27
28    def properties(self, rng, n, properties):
29        # Draw the age of onset for each individual
30        onset_age = rng.normal(self.onset_age_mean, self.onset_age_sd, size=n)
31        onset_age[onset_age < 0] = 0
32
33        # Return the calculated properties
34        return {
35            "disease_onset_age": onset_age,
36        }
37
38    def schedule_onset(self, individual):
39        # Schedule onset event
40        individual.add_to_queue_age(
41            age=individual.properties("disease_onset_age"),
42            message="disease_onset"
43        )
44
45    def onset(self, individual):
46        # Log that the disease has started
47        individual.log_event("disease_onset")
48
49        # Schedule transition to clinical state
50        individual.add_to_queue(
51            delta=self.onset_to_clinical,
52            message="disease_clinical"
53        )
54
55    def clinical(self, individual):
56        # Log the transition to the clinical state
57        individual.log_event("disease_clinical")
58
59        # Save the current age in the memory
60        individual.memory["disease_clinical_age"] = individual.age
61
62        # Schedule death event
63        individual.add_to_queue(
64            delta=self.clinical_to_death,
65            message="disease_death",
66            terminate=True
67        )

Note

The model will raise an Exception if the event tag is not added to the event_tags in the process. The same holds for durations, individual events and snapshots. Although the exception can be circumvented by setting the verify_tags parameter to False in the run() function, it is highly recommended to gather all event tags in the event_tags attribute for code clarity.

Log durations

Similarly, durations can be logged with the log_duration() method. One important difference here is that we need to specify a start_age and end_age. This is because we measure the duration between two events. In our example, it might be relevant to know how much time an individual spent in the clinical state.

We can only know how long an individual was clinical after their death. We therefore use the __end__ callback. The __end__ callback is activated once the simulation of the current individual is terminated. Let’s respond to this callback with a new method log_durations, which we add at the end of our Process.

To know at which age this individual transitioned to the clinical state, we use the disease_clinical_age value that was previously stored in the memory in line 66. Let’s name the duration we want to know disease_clinical. It starts at disease_clinical_age and ends at the current age individual.age (i.e. the age of death). The tags of each duration that can be logged by the process should be gathered in the duration_tags attribute.

When defining a callback for the __end__ message, MISCore calls this method not only with the Individual object, but also with the message that terminated the simulation. Therefore, we should also add the argument message to our method, even if it is not used in the function.

 1from miscore import Process
 2
 3
 4class Disease(Process):
 5    # All event tags that can be logged by this process
 6    event_tags = [
 7        "disease_onset",
 8        "disease_clinical",
 9        "disease_death",
10    ]
11
12    # All duration tags that can be logged by this process
13    duration_tags = [
14        "disease_clinical",
15    ]
16
17    def __init__(self, onset_age_mean, onset_age_sd, onset_to_clinical=10,
18                 clinical_to_death=5, name="disease"):
19        # Store the parameters
20        self.onset_age_mean = onset_age_mean
21        self.onset_age_sd = onset_age_sd
22        self.onset_to_clinical = onset_to_clinical
23        self.clinical_to_death = clinical_to_death
24        self.name = name
25
26        # Define how the process responds to messages
27        self.callbacks = {
28            "__start__": self.schedule_onset,
29            "disease_onset": self.onset,
30            "disease_clinical": self.clinical,
31            "__end__": self.log_durations,
32        }
33
34    def properties(self, rng, n, properties):
35        # Draw the age of onset for each individual
36        onset_age = rng.normal(self.onset_age_mean, self.onset_age_sd, size=n)
37        onset_age[onset_age < 0] = 0
38
39        # Return the calculated properties
40        return {
41            "disease_onset_age": onset_age,
42        }
43
44    def schedule_onset(self, individual):
45        # Schedule onset event
46        individual.add_to_queue_age(
47            age=individual.properties("disease_onset_age"),
48            message="disease_onset"
49        )
50
51    def onset(self, individual):
52        # Log that the disease has started
53        individual.log_event("disease_onset")
54
55        # Schedule transition to clinical state
56        individual.add_to_queue(
57            delta=self.onset_to_clinical,
58            message="disease_clinical"
59        )
60
61    def clinical(self, individual):
62        # Log the transition to the clinical state
63        individual.log_event("disease_clinical")
64
65        # Save the current age in the memory
66        individual.memory["disease_clinical_age"] = individual.age
67
68        # Schedule death event
69        individual.add_to_queue(
70            delta=self.clinical_to_death,
71            message="disease_death",
72            terminate=True
73        )
74
75    def log_durations(self, individual, message):
76        # Log durations only if individual reached the clinical state
77        if "disease_clinical_age" in individual.memory:
78            # Retrieve age at which this person reached the clinical state
79            clinical_age = individual.memory["disease_clinical_age"]
80
81            # Log the time that this individual was in the clinical state
82            individual.log_duration(
83                tag="disease_clinical",
84                start_age=clinical_age,
85                end_age=individual.age
86            )

Alternatively, durations can be logged using log_duration_start() and log_duration_end(). Modellers can use these functions to specify in their process when the duration starts or ends. The example below logs the period between onset and clinical diagnosis as disease_clinical. An error is raised when a tag supplied to log_duration_end() was never started using log_duration_start(). Any durations that were started and are not ended before the end of the simulation are ended at an individual’s death. In the example below, the duration disease_clinical starts when clinical symptoms occur. Since no instance of log_duration_end() is specified for the same tag, this duration ends at a person’s death.

 1from miscore import Process
 2
 3
 4class Disease(Process):
 5    # All event tags that can be logged by this process
 6    event_tags = [
 7        "disease_onset",
 8        "disease_clinical",
 9        "disease_death",
10    ]
11
12    # All duration tags that can be logged by this process
13    duration_tags = [
14        "disease_preclinical",
15        "disease_clinical",
16    ]
17
18    def __init__(self, onset_age_mean, onset_age_sd, onset_to_clinical,
19                 clinical_to_death, name="disease"):
20        # Store the parameters
21        self.onset_age_mean = onset_age_mean
22        self.onset_age_sd = onset_age_sd
23        self.onset_to_clinical = onset_to_clinical
24        self.clinical_to_death = clinical_to_death
25        self.name = name
26
27        # Define how the process responds to messages
28        self.callbacks = {
29            "__start__": self.schedule_onset,
30            "disease_onset": self.onset,
31            "disease_clinical": self.clinical,
32        }
33
34    def properties(self, rng, n, properties):
35        # Draw the age of onset for each individual
36        onset_age = rng.normal(self.onset_age_mean, self.onset_age_sd, size=n)
37        onset_age[onset_age < 0] = 0
38
39        # Return the calculated properties
40        return {
41            "disease_onset_age": onset_age,
42        }
43
44    def schedule_onset(self, individual):
45        # Schedule onset event
46        individual.add_to_queue_age(
47            age=individual.properties("disease_onset_age"),
48            message="disease_onset"
49        )
50
51    def onset(self, individual):
52        # Log that the disease has started
53        individual.log_event("disease_onset")
54
55        # Start a duration for the preclinical period
56        individual.log_duration_start("disease_preclinical")
57
58        # Schedule transition to clinical state
59        individual.add_to_queue(
60            delta=self.onset_to_clinical,
61            message="disease_clinical"
62        )
63
64    def clinical(self, individual):
65        # Log the transition to the clinical state
66        individual.log_event("disease_clinical")
67
68        # End duration for the preclinical period
69        individual.log_duration_end("disease_preclinical")
70
71        # Start a duration for the clinical period
72        individual.log_duration_start("disease_clinical")
73
74        # Schedule death event
75        individual.add_to_queue(
76            delta=self.clinical_to_death,
77            message="disease_death",
78            terminate=True
79        )

Log events at the individual level

In addition to the logging methods described above, you can log events at the individual level. These will appear in the events_individual DataFrame in the miscore.Result object. There are two ways to do this.

First, it is possible to log existing events at the individual level by setting log_events_individual in run() to True. Then all events that are usually logged in the events DataFrame are also logged at the individual level. You can also log a subset of the event tags at individual level by specifying a list of tags for event_individual_tags in run()

Second, you can add new individual logs to your Process using the log_event_individual() method. These logs are not logged at population level (i.e. are not added to the events) but will only appear in the events_individual DataFrame. The function takes three parameters: tag, age and element. tag specifies the tag that is logged. age indicates the age at which the event should be logged. Finally, element can be used to distinguish between events that belong to different elements of an individual (e.g. distinguish between multiple lesions). element must be an integer. Alternatively, you can use it to return a value for the individual, for example an blood level or the number of lesions the individual has.

Like with event_tags and duration_tags, you should also specify a list of all event_individual_tags at the start of the Process.

As an example, we could log the onset age of our constructed disease for each individual separately by specifying:

class Disease(Process):

    event_individual_tags = ["onset_age"]

    def schedule_onset(self, individual):
        # Schedule onset event
        individual.add_to_queue_age(
            age=individual.properties("disease_onset_age"),
            message="disease_onset"
        )
        # Log onset event at individual level
        individual.log_event_individual("onset_age")

If our disease had multiple lesions, we can use the element argument in the log_event_individual function to distinguish between the lesions. The following block of code could be added to a log_durations method to log the onset ages for each lesion at the end of a simulation.

class Disease(Process):

    event_individual_tags = ["onset_age"]

    def log_durations(self, individual):
        for i, lesion in enumerate(individual.memory["lesions"]):
            individual.log_event_individual("onset_age", element=i, age=lesion["onset_age"])

Note

Logging at individual level takes a lot of memory. Therefore you should think carefully what you want to log.