Does the Event Daemon store failed events?

Pipeline Integrations
1

We had a plug-in running on the Event Daemon. It function as desired for months, but then something happened and the plug-in was deactivated. We’re now trying to find the error and fix it, but when alterations are made to the plug-in and it’s re-loaded the Daemon spits out a long stream of what looks like previous Events that it’s trying to process but cannot.
Does it store failed Events and attempt to execute them again? And if so, is there a way to clear these stored Events to facilitate troubleshooting so we can get it working again?

2

Your Event Daemon installation config file will have a defined log location, and there will be a “shotgunEventDaemon.id” file there which holds the IDs of the plugins running.
If this is the only plugin running, you can rename the .id file to something else and restart the ED to get it running against current event IDs and not old ones.

3

Thank you! That is very useful!
However, it isn’t the only plug-in running. Is there a way to isolate one?
We may have to just reset all of 'em

4

Stop the ED, edit the .id file, and substitute a more recent event ID for the value under your plugin

5

First, Thank you. I really appreciate the assistance.
The ‘shotgunEventDaemon.id’ file is 4 lines long and consists of the names of each of the plug-ins followed by a string that looks something like this:

NUL NULˆ¹` }q†qX FF

I’m guessing that’s not what it’s supposed to look like

6

That seems really odd.
Check the eventIdFile file name and location are as specified in your shotgunEventDaemon.conf file

7

Okay. I’ve double checked the .conf file, and the .id file I’m looking at is indeed the one on the path indicated by the .conf file.
I went ahead and stopped the Daemon, renamed this file, and started it back up again.
This seems to have had no effect whatsoever. It did not generate a new .id file like the .conf file notes indicated it would and nothing seems to be working.

8

Super odd.
Are you running a current or recent version of the Event Daemon?

Returning the file to the original name should restore your ED operation, at least to the old event ID.
Maybe a different temp fix is to edit the plugin code to return right away, rather than process.
This may get you caught up to current event IDs quickly.

9

I will look into those things and get back to you!
Thank you again!

10

Things continued to degrade. We ultimately simply set up the system again, new from scratch and now we’re up and running again.
Thank you for the help!

11

Not sure if this is an advisable way to do this, but if I need to bypass or backdate the daemon, this has been my method.

I’ve added a line to shotgunEventDaemon.py that lets me set a specific event ID to start from.

def _getLastEventIdFromDatabase(self):

    conn_attempts = 0
    lastEventId = None
    while lastEventId is None:
        order = [{"column": "id", "direction": "desc"}]
        try:
            result = self._sg.find_one(
                "EventLogEntry", filters=[], fields=["id"], order=order
            )
        except (sg.ProtocolError, sg.ResponseError, socket.error) as err:
            conn_attempts = self._checkConnectionAttempts(conn_attempts, str(err))
        except Exception as err:
            msg = "Unknown error: %s" % str(err)
            conn_attempts = self._checkConnectionAttempts(conn_attempts, msg)
        else:
            #lastEventId = result["id"]
            #Swap above line to event ID that you want to start plugins off at, as per example below
            lastEventId = 12675835
            self.log.info("Last event id (%d) from the SG database.", lastEventId)
            

    return lastEventId

This script is telling the daemon to start on event 12675835.

However, this will only be pulled if the Daemon can’t find the ID file, where it logs this information. So my next step is to delete/rename the Id file, which is storing the event IDs that the script is crashing on.

This will reset things for all plugins, though, unfortunately; but this has let me be a bit more surgical, since I run into the same issue where my id file isn’t all that legible.

12

That’s an interesting solution.

Along these lines, what’s the best way to check if an entity exists given the entity ID?

All of my woes came about because a plug-in was triggered on a sequence and the sequence was deleted before the plug-in process was completed so now the plug-in was trying to run a process on an entity that didn’t exist. Because the Daemon attempts to resolve incomplete events, we ran into difficulties.

13

Ah, yeah; deleted entities were the bane of my existence awhile.

Once I’ve figured out where the error is happening, I try to deploy some error catching that allows me to log the error, but bypass that run of the script. In my experience, it usually crops up after trying to find the entity in question that the event related to.

You may require some more robust logging, maybe something that still pulls the Event ID for tracking or something, but this was an example of a simple bypass for deleted Tasks.

parent_entity = sg.find_one("Task", task_filter, ["entity"])

try:
    parent_entity_name = parent_entity["entity"]["name"]
except TypeError:
    logger.info("Task deleted; bypassed.")
    return

If it can’t find the the Entity Related to the event (and there unable to find the name of that Entity), the script will trip a Type Error, which gets put into the logs, the the “return” should help to bypass that loop.

Alternatively, you could also do an IF statement, to check if value == None, and then do the same kind of logging process.

Of course, diligence will be required; other things aside from entity deletion can cause similar errors, and you may need to expand on this error catching/logging as necessary.

1 Like
14

The .id file is encoded in cPickle format. If you want to inspect it manually, you need to load it through cPickle first … which means you need to re-pickle it if you’re going to write it back.

4 Likes
15

Thank you! That’s very good to know!