Performance profiling integration tests

Here are some little steps I take to create a query log of integration tests in Talk.

🚧 Setup

1. To speed it up and decrease setup time apply the following patch. It will make sure users are not deleted and recreated with each test which heavily improves the speed of tests in case of rerunning and also removes setup queries from your profiling:

diff --git a/tests/integration/features/bootstrap/FeatureContext.php b/tests/integration/features/bootstrap/FeatureContext.php
index 126e15258..ae89d7015 100644
--- a/tests/integration/features/bootstrap/FeatureContext.php
+++ b/tests/integration/features/bootstrap/FeatureContext.php
@@ -200,15 +200,15 @@ class FeatureContext implements Context, SnippetAcceptingContext {
 	 * @AfterScenario
 	 */
 	public function tearDown() {
-		foreach ($this->createdUsers as $user) {
-			$this->deleteUser($user);
-		}
-		foreach ($this->createdGroups as $group) {
-			$this->deleteGroup($group);
-		}
-		foreach ($this->createdGuestAccountUsers as $user) {
-			$this->deleteGuestUser($user);
-		}
+//		foreach ($this->createdUsers as $user) {
+//			$this->deleteUser($user);
+//		}
+//		foreach ($this->createdGroups as $group) {
+//			$this->deleteGroup($group);
+//		}
+//		foreach ($this->createdGuestAccountUsers as $user) {
+//			$this->deleteGuestUser($user);
+//		}
 	}
 
 	/**

2. I normally have additional files in my skeleton folder to have images etc. available when testing. In case of creating e.g. 70 users copying them takes a lot of time, so I recommend to point the skeleton either to the default directory or remove any files but the welcome.txt
3. Make sure you have a Redis/Memcache setup and use it normally and also for file locking as well:

'memcache.locking' => '\\OC\\Memcache\\Redis',

4. Run the integration test you want to profile once, so all users and groups are generated
5. Make sure you have no browser tab of the server open, no desktop client connected, xdebug disabled

🏎️ Profiling

1. Set the query_log_file in your config.php

'query_log_file' => '/var/log/nextcloud/query.log',

3. Run the integration test again
4. Save the result as baseline for your profiling (we have to clear some queries which are happening unrelated increasingly):

cat query.log | grep -v "oc_talk_internalsignaling" | grep -v "UPDATE \`oc_authtoken\`" > baseline.log

5. Clear the query log:

echo "" > query.log

6. Do some modification
7. Run the integration test again
8. Save the cleaned output as progress file:

cat query.log | grep -v "oc_talk_internalsignaling" | grep -v "UPDATE \`oc_authtoken\`" > progress.log

9. Compare the 2 query logs baseline.log and progress.log with a tool, I recommend Meld

📊 Comparing results

1. See lines dropping in Meld:
   
2. Compare query aka line counts:

cat baseline.log | wc -l
cat progress.log | wc -l

3. Compare query aka line counts for tables of your app (find a matching prefix, e.g. all talk tables start with oc_talk_):

cat baseline.log | grep "oc_talk_" | wc -l
cat progress.log | grep "oc_talk_" | wc -l

4. Compare query aka line counts for a single table in this case oc_talk_rooms:

cat baseline.log | grep "oc_talk_rooms" | wc -l
cat progress.log | grep "oc_talk_rooms" | wc -l

Start with tests/integration/features/bootstrap/FeatureContext.php and the integration test you want to profile. Follow the setup and query-log commands in the issue, including query_log_file, then determine where this profiling guidance belongs. Done means the steps are documented in the appropriate project location and can be followed to compare baseline.log with progress.log.