由于测试手法不规范造成的cts-on-gsi无法retry的问题
问题初探
测试命令： run cts-on-gsi-retry --retry 6 -s 8981083
报错堆栈(host log中)：
|
1 2 3 4 5 6 7 8 9 10 |
02-13 11:31:51 D/BaseTestSuite: Initializing ModuleRepo ABIs:[{arm64-v8a, bitness=64}] Test Args:[com.android.tradefed.testtype.AndroidJUnitTest:rerun-from-file:true, com.android.tradefed.testtype.AndroidJUnitTest:fallback-to-serial-rerun:false, com.android.compatibility.testtype.LibcoreTest:rerun-from-file:true, com.android.compatibility.testtype.LibcoreTest:fallback-to-serial-rerun:false, com.android.tradefed.testtype.AndroidJUnitTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.tradefed.testtype.HostTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.compatibility.common.tradefed.testtype.JarHostTest:exclude-annotation:android.platform.test.annotations.RestrictedBuildTest, com.android.tradefed.testtype.AndroidJUnitTest:exclude-annotation:android.platform.test.annotations.AppModeInstant, com.android.compatibility.common.tradefed.testtype.JarHostTest:exclude-annotation:android.platform.test.annotations.AppModeInstant, com.android.tradefed.testtype.HostTest:exclude-annotation:android.platform.test.annotations.AppModeInstant] Module Args:[] Includes:{armeabi-v7a CtsWindowManagerDeviceTestCases=[armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_fullscreen, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_displayCutoutNever, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_floatingWindow, armeabi-v7a CtsWindowManagerDeviceTestCases android.server.wm.LocationOnScreenTests#testLocationOnDisplay_appWindow_displayCutoutShortEdges], armeabi-v7a CtsAppSecurityHostTestCases=[armeabi-v7a CtsAppSecurityHostTestCases android.appsecurity.cts.AdoptableHostTest#testApps], armeabi-v7a CtsDevicePolicyManagerTestCases=[armeabi-v7a CtsDevicePolicyManagerTestCases com.android.cts.devicepolicy.DeviceOwnerTest#testLockTaskAfterReboot_deviceOwnerUser], armeabi-v7a CtsAccessibilityServiceTestCases=[armeabi-v7a CtsAccessibilityServiceTestCases], armeabi-v7a CtsInputMethodServiceHostTestCases=[armeabi-v7a CtsInputMethodServiceHostTestCases android.inputmethodservice.cts.hostside.InputMethodServiceLifecycleTest#testInputUnbindsOnImeStoppedFull], armeabi-v7a CtsJobSchedulerTestCases=[armeabi-v7a CtsJobSchedulerTestCases android.jobscheduler.cts.TriggerContentTest#testPhotoAdded_Reschedule], armeabi-v7a CtsNetTestCases=[armeabi-v7a CtsNetTestCases android.net.cts.DnsTest#testDnsWorks], armeabi-v7a CtsCarrierApiTestCases=[armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testUpdateConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testCarrierConfigIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testValidCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testVoicemailTableIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testQueryConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.ApnDatabaseTest#testDeleteConflictCase, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testTelephonyApisAreAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.NetworkScanApiTest#testRequestNetworkScan, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testSubscriptionInfoListing, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testVoicemailStatusTableIsAccessible, armeabi-v7a CtsCarrierApiTestCases android.carrierapi.cts.CarrierApiTest#testPhoneStateListener], armeabi-v7a CtsLocationTestCases=[armeabi-v7a CtsLocationTestCases android.location.cts.GnssPseudorangeVerificationTest#testPseudorangeValue, armeabi-v7a CtsLocationTestCases android.location.cts.GnssPseudorangeVerificationTest#testPseudoPosition, armeabi-v7a CtsLocationTestCases android.location.cts.GnssStatusTest#testGnssStatusChanges, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationValuesTest#testAccuracyFields, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRatesRepetitive, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementsConstellationTest#testGnssMultiConstellationSupported, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRatesOnOff, armeabi-v7a CtsLocationTestCases android.location.cts.GnssStatusTest#testGnssStatusValues, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementValuesTest#testListenForGnssMeasurements, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationRateChangeTest#testVariedRates, armeabi-v7a CtsLocationTestCases android.location.cts.GnssMeasurementWhenNoLocationTest#testGnssMeasurementWhenNoLocation, armeabi-v7a CtsLocationTestCases android.location.cts.GnssLocationValuesTest#testLocationRegularFields], armeabi-v7a CtsSyncContentHostTestCases=[armeabi-v7a CtsSyncContentHostTestCases], armeabi-v7a CtsStatsdHostTestCases=[armeabi-v7a CtsStatsdHostTestCases android.cts.statsd.atom.UidAtomTests#testCpuTimePerUid], armeabi-v7a CtsSecurityHostTestCases=[armeabi-v7a CtsSecurityHostTestCases android.security.cts.KernelConfigTest#testConfigROData], armeabi-v7a CtsIncidentHostTestCases=[armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.IncidentdTest#testIncidentReportDumpExplicit, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testCachedDuration, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.IncidentdTest#testIncidentReportDumpLocal, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testGpsUpdates, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testRealtime, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testJobBgVsFg, armeabi-v7a CtsIncidentHostTestCases com.android.server.cts.BatteryStatsValidationTest#testSyncBgVsFg], armeabi-v7a CtsSliceTestCases=[armeabi-v7a CtsSliceTestCases android.slice.cts.SliceMetricsTest#testLogVisible], armeabi-v7a CtsSecurityTestCases=[armeabi-v7a CtsSecurityTestCases android.security.cts.MotionEventTest#testActionOutsideDoesNotContainedObscuredInformation], armeabi-v7a CtsLibcoreTestCases=[armeabi-v7a CtsLibcoreTestCases libcore.java.nio.channels.DatagramChannelMulticastTest#test_multicastLoopOption_IPv6, armeabi-v7a CtsLibcoreTestCases libcore.java.net.SocketTest#testSocketTestAllAddresses, armeabi-v7a CtsLibcoreTestCases libcore.java.net.InetAddressTest#test_getByName_invalid, armeabi-v7a CtsLibcoreTestCases libcore.libcore.icu.DateIntervalFormatTest#testEndAtMidnight], armeabi-v7a CtsCameraTestCases=[armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraGLTest#testCameraToSurfaceTextureMetadata, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testTakePictureZsl, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testJpegExif, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAllocateBitmap, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testTakePicture, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAfRegions, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testPreviewFpsRange, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testPreviewPictureSizesCombination, armeabi-v7a CtsCameraTestCases android.hardware.cts.CameraTest#testImmediateZoom, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.ImageReaderTest#testAllOutputYUVResolutions, armeabi-v7a CtsCameraTestCases android.hardware.camera2.cts.StillCaptureTest#testAeRegions], armeabi-v7a CtsOsTestCases=[armeabi-v7a CtsOsTestCases android.os.storage.cts.StorageManagerTest#testMountAndUnmountTwoObbs, armeabi-v7a CtsOsTestCases android.os.cts.ParcelTest#testMaliciousMapWrite, armeabi-v7a CtsOsTestCases android.os.storage.cts.StorageManagerTest#testMountAndUnmountObbNormal], armeabi-v7a CtsSkQPTestCases=[armeabi-v7a CtsSkQPTestCases]} Excludes:{armeabi-v7a CtsJvmtiRunTest1906HostTestCases=[armeabi-v7a CtsJvmtiRunTest1906HostTestCases], ... } 02-13 11:31:55 D/CompatibilityTestSuite: No module that needed to run in retry were found. nothing to do. 02-13 11:31:55 I/ITestSuite: No config were loaded. Nothing to run. 02-13 11:31:55 I/ITestSuite: No config were loaded. Nothing to run. 02-13 11:31:55 I/ITestSuite: No tests to be run. |
从堆栈上看是正常的，需要再深入看一下
问题分析
首先从关键的一句log入手
No module that needed to run in retry were found. nothing to do.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
public LinkedHashMap<String, IConfiguration> loadingStrategy(Set<IAbi> abis, File testsDir, String suitePrefix, String suiteTag) { LinkedHashMap<String, IConfiguration> loadedConfigs = super.loadingStrategy(abis, testsDir, suitePrefix, suiteTag); if (loadedConfigs.size() == 0) { if (this.mIsRetry) { LogUtil.CLog.logAndDisplay(Log.LogLevel.DEBUG, "No module that needed to run in retry were found. nothing to do."); } else { throw new IllegalArgumentException(String.format("No config files found in %s or in resources.", new Object[] {testsDir .getAbsolutePath() })); } } return loadedConfigs; } |
很明显是loadingStrategy返回的LinkedHashMap为空导致的问题，那么需要看一下为什么返回值是空
看BaseTestSuite中的loadingStrategy，这里返回null
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
public LinkedHashMap<String, IConfiguration> loadingStrategy(Set<IAbi> abis, File testsDir, String suitePrefix, String suiteTag) { LinkedHashMap<String, IConfiguration> loadedConfigs = new LinkedHashMap(); if (!this.mSkipJarLoading) { loadedConfigs.putAll( getModuleLoader().loadConfigsFromJars(abis, suitePrefix, suiteTag)); } if (this.mConfigPatterns.isEmpty()) { this.mConfigPatterns.add(".*\\.config"); this.mConfigPatterns.add(".*\\.xml"); } loadedConfigs.putAll( getModuleLoader() .loadConfigsFromDirectory(testsDir, abis, suitePrefix, suiteTag, this.mConfigPatterns)); return loadedConfigs; } |
这个函数是为了加载我们给测试项提供的额外的配置文件，一般在google/vts/vts9.0_r5/android-vts/testcases的文件下，如BionicUnitTestsStatic.config等等，从中读取配置文件；retry时其调用栈为：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
vts-tf > java.lang.RuntimeException at com.android.tradefed.testtype.suite.BaseTestSuite.loadingStrategy(BaseTestSuite.java:91) at com.android.compatibility.common.tradefed.testtype.suite.CompatibilityTestSuite.loadingStrategy(CompatibilityTestSuite.java:164) at com.android.tradefed.testtype.suite.BaseTestSuite.loadTests(BaseTestSuite.java:76) at com.android.compatibility.common.tradefed.testtype.suite.CompatibilityTestSuite.loadTests(CompatibilityTestSuite.java:113) at com.android.tradefed.testtype.suite.ITestSuite.loadAndFilter(ITestSuite.java:262) at com.android.tradefed.testtype.suite.ITestSuite.createExecutionList(ITestSuite.java:304) at com.android.tradefed.testtype.suite.ITestSuite.run(ITestSuite.java:367) at com.android.compatibility.common.tradefed.testtype.retry.RetryFactoryTest.run(RetryFactoryTest.java:177) at com.android.tradefed.invoker.InvocationExecution.runTests(InvocationExecution.java:384) at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:358) at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:205) at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:667) at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:566) |
继续沿着栈查看
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
public LinkedHashMap<String, IConfiguration> loadConfigsFromDirectory(File testsDir, Set<IAbi> abis, String suitePrefix, String suiteTag, List<String> patterns) { LinkedHashMap<String, IConfiguration> toRun = new LinkedHashMap(); List<File> listConfigFiles = new ArrayList(); List<File> extraTestCasesDirs = Arrays.asList(new File[] { testsDir }); listConfigFiles.addAll( ConfigurationUtil.getConfigNamesFileFromDirs(suitePrefix, extraTestCasesDirs, patterns)); Collections.sort(listConfigFiles); for (File configFile : listConfigFiles) { toRun.putAll( loadOneConfig(configFile .getName(), configFile.getAbsolutePath(), abis, suiteTag)); } return toRun; } |
可见有两种可能 1. listConfigFiles为空 2. 循环loadOneConfig一直没有加载config
若第一种情况，那么只能是测试环境有问题，配置文件完全被删了，看了下排除该情况；那么只能是第二种情况
查看SuiteModuleLoader中的相应逻辑
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
private LinkedHashMap<String, IConfiguration> loadOneConfig(String configName, String configFullName, Set<IAbi> abis, String suiteTag) { LinkedHashMap<String, IConfiguration> toRun = new LinkedHashMap(); String name = configName.replace(".config", ""); String[] pathArg = { configFullName }; try { for (IAbi abi : abis) { String id = AbiUtils.createId(abi.getName(), name); if (shouldRunModule(id)) { IConfiguration config = this.mConfigFactory.createConfigurationFromArgs(pathArg); if (!Strings.isNullOrEmpty(suiteTag)) { if (!config.getConfigurationDescription().getSuiteTags().contains(suiteTag)) { LogUtil.CLog.d("Configuration %s does not include the suite-tag '%s'. Ignoring it.", new Object[] { configFullName, suiteTag }); continue; } } List<ConfigurationDef.OptionDef> optionsToInject = new ArrayList(); if (this.mModuleOptions.containsKey(name)) { optionsToInject.addAll((Collection)this.mModuleOptions.get(name)); } if (this.mModuleOptions.containsKey(id)) { optionsToInject.addAll((Collection)this.mModuleOptions.get(id)); } config.injectOptionValues(optionsToInject); ITargetPreparer preparer; List<ITargetPreparer> preparers = config.getTargetPreparers(); for (Iterator localIterator2 = preparers.iterator(); localIterator2.hasNext();) { preparer = (ITargetPreparer)localIterator2.next(); if ((preparer instanceof IAbiReceiver)) { ((IAbiReceiver)preparer).setAbi(abi); } } Object tests = config.getTests(); for (IRemoteTest test : (List<IRemoteTest>)tests) { String className = test.getClass().getName(); if (this.mTestOptions.containsKey(className)) { config.injectOptionValues((List)this.mTestOptions.get(className)); } addFiltersToTest(test, abi, name, this.mIncludeFilters, this.mExcludeFilters); if ((test instanceof IAbiReceiver)) { ((IAbiReceiver)test).setAbi(abi); } } config.getConfigurationDescription().setAbi(abi); config.getConfigurationDescription().setModuleName(name); toRun.put(id, config); } } } catch (ConfigurationException e) { throw new RuntimeException(String.format("Error parsing configuration: %s: '%s'", new Object[] { configFullName, e .getMessage() }), e); } return toRun; } |
|
1 2 3 4 5 6 |
private boolean shouldRunModule(String moduleId) { List<SuiteTestFilter> mdIncludes = getFilterList(this.mIncludeFilters, moduleId); List<SuiteTestFilter> mdExcludes = getFilterList(this.mExcludeFilters, moduleId); return ((this.mIncludeAll) || (!mdIncludes.isEmpty())) && (!containsModuleExclude(mdExcludes)); } |
我们注意到在loadOneConfig中有个过滤条件，shouldRunModule；那么在retry时这里的逻辑是过滤出要被执行的module的相应的配置属性
那么为什么这个值一直是false呢，mIncludeFilters本身不是null（host log中打印出来了），那么只能getFilterList(this.mIncludeFilters, moduleId)得到null；
moduleId为abi和module name的组合，到这里，我们就怀疑是abi造成的问题了；再回头看host log；发现果然：
|
1 2 |
ABIs:[{arm64-v8a, bitness=64}] Includes:{armeabi-v7a CtsWindowManagerDeviceTestCases=[armeabi-v7a CtsWindowManagerDeviceTestCases ... |
由于abi的不匹配造成的问题，retry时查询要执行的module的config时，由于abi不匹配造成测试框架误判断shouldRunModule = false；
问题总结
1.测试手法的问题，cts-og-gsi默认只测64位，但是如果后面加上-a armeabi-v7a的话，也能进行32位的测试；测试第一次测试时加了-a armeabi-v7a；但是后面进行retry时没有加，导致框架默认abi为arm64-v8a；导致后续retry流程module因为abi的关系无法匹配；读取要retry的报告为v7a，加载config时默认v8a导致的问题， 请测试同学注意下；测cts-on-gsi时后面就不用加-a了，避免该问题重复发生
2.框架有轻微的问题，但本质来说是测试手法问题，google不会认的