Too much contention on these documents #856

Activity

1. sshahdev

   added

   api: firestore

   on Apr 17, 2020
2. sshahdev

   assigned

   schmidt-sebastian

   on Apr 17, 2020
3. google-oss-bot

   added

   needs-triage

   on Apr 17, 2020
4. google-oss-bot commented on Apr 17, 2020

   

   google-oss-bot

   on Apr 17, 2020

   More actions

   I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
5. schmidt-sebastian commented on Apr 17, 2020

   

   schmidt-sebastian

   on Apr 17, 2020

   Contributor

   More actions

   Please see the comment here: googleapis/nodejs-firestore#1020

   Thank you!
6. schmidt-sebastian

   closed this as completedon Apr 17, 2020
7. schmidt-sebastian commented on Apr 17, 2020

   

   schmidt-sebastian

   on Apr 17, 2020

   Last edited by schmidt-sebastian

   Contributor

   More actions

   @sshahdev Can you send me your project ID to mrschmidt(at)google.com? Our backend team wants to take a closer look.
8. schmidt-sebastian commented on Apr 17, 2020

   

   schmidt-sebastian

   on Apr 17, 2020

   Contributor

   More actions

   Internal bug: b/154364083
9. dooleyb1 commented on Apr 23, 2020

   

   dooleyb1

   on Apr 23, 2020

   More actions

   I am having this issue in a production environment, we sent out a notification to >3000 users which would have triggered an API request to access a specific document. From looking through the logs, this error caused some of our users (~130+ from what I can see) requests to fail as a result of this error which prevented them from carrying out expected functionality within our application.

   @schmidt-sebastian I can provide further information to you if needs be for investigation.
10. schmidt-sebastian commented on Apr 24, 2020

    

    schmidt-sebastian

    on Apr 24, 2020

    Contributor

    More actions

    @dooleyb1 Did you write to this document at the same time or had a transaction open that held a lock on this document (by calling Transaction.get())?
11. dooleyb1 commented on Apr 24, 2020

    

    dooleyb1

    on Apr 24, 2020

    More actions

    @schmidt-sebastian Yes, the document represents a survey that users are trying to start simultaneously. We update a counter on the document to the number of users currently doing the survey as we have an upper limit on the number of users who can complete a given survey so want to prevent access to the document once this number has been reached. From reading transaction documentation I believed this was how I should handle such a use case.

    A simple version of the transaction is as follows:

    firestore.runTransaction(transaction => {
        return transaction.get(surveyRef)
            .then(async surveyDoc => {
                let surveyData = surveyDoc.data();
    
                // Get the availableQuotaPaths for user
                const userDoc = await firestore.collection('users').doc(uid).get();
                const usersEligibleQuotaPaths = await getUsersQuotaPathsForBundle(userDoc.data(), surveyDoc);
    
                // If the user has no availableQuotaPaths, don't let them start
                if (usersEligibleQuotaPaths.length <= 0) {
                    return Promise.reject(`Capacity not available for bundle ${surveyId} for user ${uid}`);
                }
    
                // If there is quota paths for the user, select the one with highest weight and return questions
                let currentBestWeight = 0;
                let optimalQuotaPath = null;
    
                for (let index in usersEligibleQuotaPaths) {
                    let quotaPathToCheck = usersEligibleQuotaPaths[index];
    
                    let weight = surveyData['quotas'][quotaPathToCheck]['weight'];
    
                    // If better weight at this quotaPath, update selection
                    if (weight > currentBestWeight) {
                        optimalQuotaPath = quotaPathToCheck;
                        currentBestWeight = weight;
                    } 
                }
    
                // Update the quota count for quotaPath
                transaction.update(surveyRef, {
                    [`userQuotaPathMap.${uid}`]: optimalQuotaPath,
                    [`quotas.${optimalQuotaPath}.currentQuota`]: admin.firestore.FieldValue.increment(1),
                    [`quotas.${optimalQuotaPath}.usersInBundle`]: admin.firestore.FieldValue.increment(1),
                    usersInBundle: admin.firestore.FieldValue.increment(1)
                });
    
                // Return the questions
                const surveyQuestionCollection = await firestore.collection('activeSurveys').doc(surveyId).collection('questions').get();
                let surveyQuestions = {};
    
                surveyQuestionCollection.docs.forEach(questionDoc => {
                    surveyQuestions[questionDoc.id] = questionDoc.data();
                });
    
                return Promise.resolve(surveyQuestions);
            })
    }).then(surveyQuestions => {
        // Handle transaction result here
    })
12. schmidt-sebastian commented on Apr 24, 2020

    

    schmidt-sebastian

    on Apr 24, 2020

    Contributor

    More actions

    A transactional lock on a document can cause contention errors to show up on reads for the document, since the read needs to access the same locked resource. Can you use a different document for your counter?
13. dooleyb1 commented on May 6, 2020

    

    dooleyb1

    on May 6, 2020

    Last edited by dooleyb1

    More actions

    A transactional lock on a document can cause contention errors to show up on reads for the document, since the read needs to access the same locked resource. Can you use a different document for your counter?

    Yes I could use a different document but I'm not sure if this will solve my issue. It is likely that I will need to update this counter more than once per second as users open push notifications and from reading around this seems to be the update limit on Firebase is this correct? If so, how should one handle the necessity to update a document more than once per second?
14. varun-git28 commented on May 7, 2020

    

    varun-git28

    on May 7, 2020

    More actions

    Im also getting this error.
    { Error: Too much contention on these documents. Please try again.
    at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/client.js:101:45)
    at emitOne (events.js:121:20)
    at Http2CallStream.emit (events.js:211:7)
    at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
    at _combinedTickCallback (internal/process/next_tick.js:132:7)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9)
    code: 10,
    details: 'Too much contention on these documents. Please try again.',
    metadata: Metadata { options: undefined, internalRepr: Map {} } }