AI Error Handling : database and interface

[hbakhshi]

The request is to store the suggestion of AI for each new action in the database. There could be more than one suggestion per action.

In addition to the database modification, we need to implement a system to run the codes and extract the suggestions. I can help to implement this part, if I know how it should work.

[vargasa]

@hbakhshi Here you can find an example of how the models are defined:

wtc-console/src/workflows/models.py

Line 36 in 4157709

class TaskActionParameters(EmbeddedDocument):

We can make it a Django app to handle the backend or if it is not well defined we can make it part of misc

https://github.com/CMSCompOps/wtc-console/tree/master/src/misc

[hbakhshi]

Thanks @vargasa . I will look at them and try to find how we should put our tools in.
Here is some details about what we need.

For the database : We need one table with the following fields :

1. TasksID : one-to-many linked to the table of tasks
2. ModelID (Integer) : an integer showing the ID of the model
3. InputTrainingDatasetID (Integer) : to store the information about the input training dataset
4. PredictedAction : the type and allowed values should be the same as the action by the operator (acdc/clone/.....)
5. Significance (float) : to store the estimated significance of the prediction
6. Details (Text) : a placeholder to store any additional information
   ---- TaskID, ModelID, InputTrainingDatasetID is the primary key or should set to be unique.

For the API : We will develop the API that can be called and it returns the predictions in a format compatible with the table format to be inserted in the database. This output can be also interpreted by the console to be presented to the operator at some point.
a. We need to know the format of the input data that can be passed to the API for each task.
b. We need to know if the console can provide the log files or their addresses to us.

[vargasa]

@hbakhshi

a. We need to know the format of the input data that can be passed to the API for each task.

wtc-console/src/workflows/models.py

Line 36 in 4157709

class TaskActionParameters(EmbeddedDocument):

b. We need to know if the console can provide the log files or their addresses to us.

If these information is needed I'd rather think about making the whole thing an app within this console, that would probably save us time and you'll have access to the databases wtc has access to. @dabercro What do you think?

[dabercro]

I don't think the models need to be made inside this console, but they will need to have a uniform interface. Maybe this console can define a base class or even have a simple implementation, but that's not strictly necessary as long as everyone agrees on object methods and signatures.

That way, as far as this console is concerned, all of the models can just be a list of objects with the same methods. This console can import the model packages as needed and provide the connection between the databases and the methods.

[vlimant]

the model prediction should not be run on the console VM, so that we have a structure that is flexible and can accommodate for multiple engines (which might in the end require more than what a single VM can provide)

what needs to be defined IMO, is the schema/location of the suggestions that an external engine can push in 274-mongodb, and how it gets visualized/used on the console (related to #7 somehow)

[hbakhshi]

I have just committed the first version of the AIErrorHandling on my github here :
https://github.com/hbakhshi/AIErrorHandling/
API to get all the AI suggestions can be found in the models sub-package.

It needs tensorflow 1.13.
it has been tested and is compatible with python 2.7 and 3.5
we can try to install it on the new-console machine and see how it works.

There is a place to add the codes for training. I have already added my codes under training/SitesErrorCodes. @llayer can also add his training codes under the training sub-package.

[dabercro]

I copied the repository for you here: https://github.com/CMSCompOps/AIErrorHandling. @hbakhshi you should have admin access after you accept the invitation.

In my experience, I used conda to install tensorflow on one of the CERN VMs (miniconda was enough: https://conda.io/en/latest/miniconda.html). Hopefully it's straightforward to get Django to play nice with it too.