New objects not getting added to parent object - couchdb

[AdamElliott-AB]

Summary

We are using couchdb for persistence. Recently started having problems creating new objects. New objects seem to be created fine at first. They can be viewed and edited, but if the page is refreshed the new object is gone. Looking at the network traffic and couchdb database revealed that new object documents are getting added to couchdb, but the parent object document is not getting updated to include the new child object. No errors were reported on the browser console or couchdb logs.

Troubleshooting

1. Problem persisted after using the couchdb admin tool to delete all documents except "mine" and updating "mine" to have an empty composition array.
2. Problem was resolved after fully deleting the couchdb openmct database and creating a new empty openmct database. OpenMCT created a new "mine" document when the first openmct object was added.

Workaround

1. export original "My Items" as json
2. delete couchdb openmct database
3. create new empty openmct database
4. In OpenMCT create new folder under "My Items" called "clone". This initializes the new openmct database.
5. Import json file into the "clone" folder
6. Move objects under the "clone/My Items" folder to be directly under the root "My Items" folder
7. Delete the "clone" folder

Steps to Reproduce

1. We are running multiple instances of openmct/couchdb with the same configuration and environment. The only difference is the OpenMCT views/objects that users have created. The problem was only observed on one instance. This instance had over 200 documents in couchdb which included several orphan objects after the onset of the problem.

Environment

The problem was observed using openmct versions 1.7.7 and 1.7.5

[shefalijoshi]

@AdamElliott-AB Which version of Open MCT were you using before you updated to 1.7.7/1.7.5?

[AdamElliott-AB]

We were using 1.7.5. We did not notice the problem until we updated to 1.7.7 We afterwards reverted to 1.7.5 but the problem was still present.

[akhenry]

@AdamElliott-AB A couple of questions:

1. Is it possible that multiple users were working with the parent object at the same time? (eg. multiple people creating objects under the same folder, including the "My Items" folder). This sounds like the behavior when a Couch conflict occurs (HTTP response code 409). Basically if multiple people try to create objects under the same parent object the new objects get created just fine, but updating the composition of the parent fails because one of the parties is trying to update a stale model. So you end up with orphaned objects. This should show a persistence error notification in the UI though. We are working to reduce the number of situations in which a user is editing a stale object model, but currently it's still easy to do with folders.
2. Is Couch sitting behind a reverse proxy? Is it authenticated? Could this be the same authentication issue that was affecting telemetry requests?

[AdamElliott-AB]

@akhenry

1. Nope. Every newly created document had this problem, regardless of location on the object tree. Was seen multiple times with only one user accessing the system
2. Couch is behind a reverse proxy.It was not an authentication problem though. I watched the network activity in chrome during object creation. No unauthorized errors. OpenMCT did a PUT for the new document, then a GET of the new document, both of which were successful, but never proceeded to do a PUT on the parent document (which should be the next step based on my observation of other working installations). I also verified with couchdb admin that the new object document did indeed get stored in couchdb as expected.

[akhenry]

@AdamElliott-AB Thanks for the info. Which version of Couch are you using?

We are not seeing exactly what you're describing in our own deployments, but we are tracking an issue with some similarities. The issue we have observed is transient though and we haven't managed to reproduce it reliably.

[AdamElliott-AB]

couchdb v 3.1.1

The issue we saw was consistent once it started. It continued after server restart, reverting to an earlier version of OpenMCT, and even deleting all couch documents. That last one I found particularly strange as it seems to indicate that OpenMCT's interaction with couch is somehow affected by deleted documents. I did back up the problematic database before completely wiping it out. I think I can get that isolated to a shareable state if interested.

[akhenry]

@AdamElliott-AB Is it possible that your browser or proxy have cached 1.7.7, and that after you reverted to 1.7.5 you were actually still using 1.7.7? This can happen sometimes when switching versions. You can confirm the version by clicking on the Open MCT logo in the top right.

Also, are you seeing any errors when this problem occurs? In the network tab, in the JavaScript console, or in the UI?

[AdamElliott-AB]

I hadn't considered caching after the version change. I doubt it though as I have browser cache disabled setting whenever browser dev tools is open, and I had that open frequently when troubleshooting. I did not see any errors in the server log, javascript console or the UI. I did not notice any errors in the network tab either, although I was not monitoring that all of the time.

[shefalijoshi]

I was able to reproduce similar behavior by doing the following:

1. Create an empty openmct database in CouchDB
2. Right-click on My Items and choose 'Add new Folder'.
3. Click OK. You will see a persistence error:
   
4. Notice that the Folder DOES show up in the navigation tree.
5. Import a few objects into this folder using JSON.
6. Edit these objects and save them
7. Refresh the browser
8. Boom - gone!

I was not able to reproduce disappearing items by Creating them from the create menu. I had to import them.

[AdamElliott-AB]

We just ran into this again. I still have not been able to identify the trigger, but once OpenMCT gets into this state it stays that way until I delete the couchdb 'openmct' database and re-import our objects from json into a fresh database as described earlier.

[shefalijoshi]

@AdamElliott-AB Thanks for the update.
I think it might help to see your workflow so that we can try to reproduce this in a similar way that you do. We will get in touch with you soon.

[shefalijoshi]

This is happening because the parent composition does not get updated and saved.
The save does not happen because the persisted time stamp of the objects is > the modified timestamp of the object.
(Thus concluding that the object has not been modified on this line: https://github.com/nasa/openmct/blob/1.7.8/platform/core/src/runs/TransactingMutationListener.js#L40)

[shefalijoshi]

Closing this as an environment specific problem.