Android I/O Prefetch 学习

2023-09-23 52 0

Android Iop 学习

思路整理:明确方向和目的

这个季度(剩余60天),我有一个重要的任务:完成Iop(I/O Prefetcher)的原理学习。目前我已经看Iop快一周了,但是还是不怎么懂,感觉距离理解还得一个多月,最主要的原因是其所涉及到的内容很广,从Android 的 FrameWork层延申到了hal层,自顶向下的链路大概是:
FrameWork{AMS,BoostFrameWork(通过反射调用Performance)}Java语言实现 ----> 中间转发层 Preformance 通过 jni ----> 调用Hal 层的 c++ 代码,c++ 代码完成预取操作。
这个学习过程有一个难点的地方是,AMS,这个地方所涉及的知识很多,又比较杂,BoostFrameWork穿插在AMS这个框架里的不少地方,来实现Iop。而AMS(ActivityManagerService)这个是Android 四大组件中最重要的一个,其功能主要是用来管理进程的启动调度,及内存管理。
在这个季度尾,我们部门有一个一小时的学习内容讲解,需要有深度的讲解一下技术,我师傅对我的安排是把Iop输出文档,并对Iop做一个有深度的讲解,目前看来,我的心里是没什么把握的,毕竟网上几乎搜不到相关的内容,唯一一篇文章,android IO Prefetch源码分析,很多地方讲的比较粗,看起来很费劲,对于我一个Android系统 新手来说,在FrameWork层的理解还是新手,不过这篇文章也很有参考意义。现在来罗列一下我目前所拥有的工具与资料:

  1. android IO Prefetch 源码分析
  2. Android 11 Iop HAL层 + BoostFrameWork源码框架 BoostFramework未穿插在 FrameWork 中
  3. Android 10 Iop 在FrameWork 层的源码,BoostFramework 穿插在 FrameWork中
  4. 随处可见的 AMS 讲解资料
  5. 调试的板子和环境可用于验证结果

我的任务:

  1. 完成 Iop 在 Android 11 的成功启动
  2. 测试 Iop 在 App 冷启动方面的优化效果
  3. 确认是否集成到新平台

标题学习计划制定

第一阶段

以目前的状态,自顶向下的学习,是效率最高的一种方案。故采取自顶向下的学习方式,来进行学习。虽说自顶向下,但是并非需要将FrameWork全学了,这样时间肯定不足,也大大脱离了目标。对于上层的学习目前只需要学习AMS,但是AMS依然很大,由我最近这几天的学习,发现AMS依然可以拆分,大致分为:

ActivityManagerService.java
ActivityStack.java
ProcessRecord.java
ActivityStackSupervisor.java
ActivityStarter.java
ActivityRecord.java

以上这几个模块需要重点关注,其余知识大致理解流程即可,以上这几个模块的代码重点阅读与理解,毕竟要在Android 11 上对这几块代码进行修改,现在拥有Android 10 编写的Iop的框架,可以进行模仿,但是前提条件是需要理解这几部分是做什么的,不然所做的代码移植就是不经思考的模仿。

第一阶段结果展望:

预期日期 截止日期 结果
2022/11/4 2022/11/10 完成Iop的成功启动,且输出一篇BoostFrameWork Iop部分和AMS部分互相作用机制的文档。
Activity 启动流程梳理

下图是Activity启动的框架流程图,包含了activity 整体流程的框架
在这里插入图片描述
较老Android 版本 Activity启动流程图
在这里插入图片描述
手动绘制的代码流程图,Activity的启动流程
在这里插入图片描述

  1. 学习Iop中遇到的一个人让我疑惑不解问题,I/O Prefetch 在AMS中是在哪些节点上生效会导致App启动被加速?
    根据上图可知道Activity的启动会涉及两种情况,第一种是已经被启动过了一次,可以直接调用realStartActivityLocked()来启动,还有一种是未曾被启动过,需要调用startProcessAsync通过层层调用使用socket通知Zygote去fork一个新的进程,但最后通过层层调用还是会回到realStartActivityLocked来启动,而realStartActivityLocked 调用了findTaskToMoveToFront,而其包含了acquireAppLaunchPerfLock,acquireAppLaunchPerfLock 就是我们执行I/O Prefetch的部分,在此过程中发生Iop预读来加速App启动时间。

    android/frameworks/base/services/core/java/com/android/server/wm/ActivityStackSupervisor.java

void startSpecificActivity(ActivityRecord r, boolean andResume, boolean checkConfig) {...........boolean knownToBeDead = false;if (wpc != null && wpc.hasThread()) {try {realStartActivityLocked(r, wpc, andResume, checkConfig);return;} catch (RemoteException e) {Slog.w(TAG, "Exception when starting activity "+ r.intent.getComponent().flattenToShortString(), e);}.......final boolean isTop = andResume && r.isTopRunningActivity();mService.startProcessAsync(r, knownToBeDead, isTop, isTop ? "top-activity" : "activity");}
void findTaskToMoveToFront(Task task, int flags, ActivityOptions options, String reason,boolean forceNonResizeable) {.....//top_activity = task.stack.topRunningActivityLocked();/* App is launching from recent apps and it's a new process */ActivityRecord top_activity = task.topRunningActivityLocked();if(top_activity != null && top_activity.isState(DESTROYED)) {acquireAppLaunchPerfLock(top_activity.packageName);}....

这里使用了BoostFramework 里面的调用,开启I/O预取,但并非第一次启动App

public void acquireAppLaunchPerfLock(String packageName) {/* Acquire perf lock during new app launch */if (mPerfBoost == null) {mPerfBoost = new BoostFramework();}if (mPerfBoost != null) {mPerfBoost.perfHint(BoostFramework.VENDOR_HINT_FIRST_LAUNCH_BOOST, packageName, -1, BoostFramework.Launch.BOOST_V1);mPerfSendTapHint = true;}if (mPerfPack == null) {mPerfPack = new BoostFramework();}if (mPerfPack != null) {mPerfPack.perfHint(BoostFramework.VENDOR_HINT_FIRST_LAUNCH_BOOST, packageName, -1, BoostFramework.Launch.BOOST_V2);}// Start IOPif (mPerfIop == null) {mPerfIop = new BoostFramework();}if (mPerfIop != null) {mPerfIop.perfIOPrefetchStart(-1,packageName,"");}}public void acquireUxPerfLock(int opcode, String packageName) {mUxPerf = new BoostFramework();if (mUxPerf != null) {mUxPerf.perfUXEngine_events(opcode, 0, packageName, 0);}}
  1. 第二个问题,为什么在次节点上会产生加速效果?
    原因:许多应用程序在启动时需要访问I/O.很多时间会因为阻塞I / O而导致应用程序启动慢。预取数据之后,应用程序几乎可以从pagecache 中立即访问该数据,从而大大减少了应用程序启动延迟。

接下来要聊的是Activity 的状态转换关系。
在这里插入图片描述
基于上面的结论,我们启动一个新的App会调用到realStartActivityLocked这个函数,而此时Activity的状态正是处于OnCreate开始前,处于INITIALIZING 状态,在此处我们插入Iop的预取可实现优化。Activity 的所有状态如下:

    enum ActivityState {INITIALIZING,RESUMED,PAUSING,PAUSED,STOPPING,STOPPED,FINISHING,DESTROYING,DESTROYED,RESTARTING_PROCESS}

整体流程节点如下:
在这里插入图片描述

完成日期 完成结果
2022/10/30 梳理了Activity的启动及IO Prefetch的执行节点
第二阶段

第二阶段相对第一阶段较容易,因为这一周的时间已经用来看了Iop的hal层的实现机制了,且代码上的分布很紧凑,代码量小,涉及的架构不怎么庞大,正常进行学习即可,顺便学习一下JNI机制,衍射。
下图是关于Iop调用的整体流程图如下:
在这里插入图片描述
Iop初始化流程
首先当系统启动时,会启动 vendor.qti.hardware.iop@2.0-service 进程,该进程相当于一个Server端,他在最开始运行时会进行初始化,首先调用dlopen加载libqti-iopd.so动态库,初始化函数指针,加载libqti-iopd.so库中的函数,以便之后用到的时候,对函数进行调用。接下来看看他是如何做的:

void Iop::LoadIopLib() {const char *rc = NULL;char buf[PROPERTY_VALUE_MAX];if (!mHandle.is_opened) {mHandle.dlhandle = dlopen("libqti-iopd.so", RTLD_NOW | RTLD_LOCAL);if (mHandle.dlhandle == NULL) {ALOGE("IOP-HAL %s Failed to (dl)open iopd\n", __func__);return;}dlerror();*(void **) (&mHandle.iop_server_init) = dlsym(mHandle.dlhandle, "iop_server_init");if ((rc = dlerror()) != NULL) {ALOGE("IOP-HAL %s Failed to get iop_server_init\n", rc);dlclose(mHandle.dlhandle);mHandle.dlhandle = NULL;return;}*(void **) (&mHandle.iop_server_exit) = dlsym(mHandle.dlhandle, "iop_server_exit");if ((rc = dlerror()) != NULL) {ALOGE("IOP-HAL %s Failed to get iop_server_exit\n", rc);dlclose(mHandle.dlhandle);mHandle.dlhandle = NULL;return;}*(void **) (&mHandle.iop_server_submit_request) = dlsym(mHandle.dlhandle, "iop_server_submit_request");if ((rc = dlerror()) != NULL) {ALOGE("IOP-HAL %s Failed to get iop_server_submit_request\n", rc);dlclose(mHandle.dlhandle);mHandle.dlhandle = NULL;return;}*(void **) (&mHandle.uxe_server_submit_request) = dlsym(mHandle.dlhandle, "uxe_server_submit_request");if ((rc = dlerror()) != NULL) {ALOGE("IOP-HAL %s Failed to get uxe_server_submit_request\n", __func__);dlclose(mHandle.dlhandle);mHandle.dlhandle = NULL;return;}mHandle.is_opened = true;}return;
}

在加载了libqti-iopd.so动态库的函数之后,该进程便会调用iop_server_init的初始化操作,在这里调用了iop_server_init等待Framework层传下来的信息进行处理。

IIop* HIDL_FETCH_IIop(const char* /* name */) {ALOGE("IOP-HAL: inside HIDL_FETCH_IIop");Iop *iopObj = new (std::nothrow) Iop();ALOGE("IOP-HAL: boot Address of iop object");if (iopObj != NULL) {iopObj->LoadIopLib();ALOGE("IOP-HAL: loading library is done");if (iopObj->mHandle.iop_server_init != NULL ) {(*(iopObj->mHandle.iop_server_init))();}}return iopObj;
}

Iop_server_init这个函数会创建了一个死循环的iop_server线程,用来处理从Framework层传下来的msg信息。

int iop_server_init() {int rc1 = 0, stage = 0, rc2 = 0;int uba_rc = 0;char property[PROPERTY_VALUE_MAX];char trace_prop[PROPERTY_VALUE_MAX];QLOGI("IOP server starting");/* Enable traces by adding vendor.debug.trace.perf=1 into build.prop */if (property_get(PROP_NAME, trace_prop, NULL) > 0) {if (trace_prop[0] == '1') {perf_debug_output = PERF_SYSTRACE = atoi(trace_prop);}}preferred_apps = (char *) malloc (strcat_sz * sizeof(char));if (preferred_apps == NULL) {goto error;}memset(preferred_apps, 0, strcat_sz);preferred_apps[0] = '\0';IOPevqueue.GetDataPool().SetCBs(Alloccb, Dealloccb);rc1 = pthread_create(&iop_server_thread, NULL, iop_server, NULL);if (rc1 != 0) {stage = 3;goto error;}iop_init = true;if (uxe_disabled()) {uxe_prop_disable = 1;} else {uxe_prop_disable = 0;strlcpy(property,perf_get_prop("vendor.iop.uxe_trigger_freq" , "1").value, PROPERTY_VALUE_MAX);property[PROPERTY_VALUE_MAX-1]='\0';uxe_trigger_freq = atoi(property);UXEevqueue.GetDataPool().SetCBs(Alloccb, Dealloccb);rc2 = pthread_create(&uxe_server_thread, NULL, uxe_server, NULL);uba_rc = init_uba();if (rc2 != 0) {goto error;}if (uba_rc == 0) {QLOGE("Fail to init UBA");goto error;}uxe_init = true;}return 1;error:QLOGE("Unable to create control service (stage=%d, rc1=%d rc2=%d uba_rc=%d)", stage, rc1, rc2, uba_rc);return 0;
}

在 iop_server中,首先初始化一些参数,然后会向IOPevqueue队列申请一个msg,若队列中没有请求消息则阻塞在这里,等待队列中出现msg,若有msg,则取出来,走接下来的流程,然后创建数据库,若数据库没有被创建则创建数据库。因为我们提交的消息一般是IopStart 的过程,所以消息类型是case IOP_CMD_PERFLOCK_IOPREFETCH_START,所以我们走这条分支。接下来他会获取配置文件的配置参数,是否开启某些功能。配置文件路径 /vendor/etc/perf/perfconfigstore.xml ,配置文件基本内容如下图
在这里插入图片描述
可根据vendor.enable.prefetch是否为1是否开启iop预读。最开始执行信息采集的是pid > 0,所以先分析pid > 0的情况。

static void *iop_server(void *data)
{int rc, cmd;iop_msg_t *msg = NULL;(void)data;bool is_db_init = false;char tmp_pkg_name[PKG_NAME_LEN];int bindApp_dur = 0, disAct_dur = 0, avg_bindApp = 0, avg_disAct = 0;int pkg_count = 0, bindApp_count = 0, pkg_mismatch = 0, da_reset = 0;bool launching = false;memset(tmp_pkg_name, 0, PKG_NAME_LEN);/* Main loop */for (;;) {//wait for perflock commandsEventData *evData = IOPevqueue.Wait();if (!evData || !evData->mEvData) {continue;}if(!is_boot_complete()){QLOGE("io prefetch is disabled waiting for boot_completed");continue;}if(is_db_init == false){if(create_database() == 0){//Successis_db_init = true;}}case IOP_CMD_PERFLOCK_IOPREFETCH_START:{static bool is_in_recent_list = false;char enable_prefetch_property[PROPERTY_VALUE_MAX];char enable_prefetch_ofr_property[PROPERTY_VALUE_MAX];int enable_prefetcher = 0;int enable_prefetcher_ofr = 0;strlcpy(enable_prefetch_property,perf_get_prop("vendor.enable.prefetch" , "0").value, PROPERTY_VALUE_MAX);enable_prefetch_property[PROPERTY_VALUE_MAX-1]='\0';enable_prefetcher = strtod(enable_prefetch_property, NULL);strlcpy(enable_prefetch_ofr_property,perf_get_prop("vendor.iop.enable_prefetch_ofr" , "0").value, PROPERTY_VALUE_MAX);enable_prefetch_ofr_property[PROPERTY_VALUE_MAX-1]='\0';enable_prefetcher_ofr = strtod(enable_prefetch_ofr_property, NULL);// if PID < 0 consider it as playback operationif(msg->pid < 0){int ittr = 0;char *week_day = NULL;week_day = (char *) malloc(6*sizeof(char));// Insert package into the tableif (week_day == NULL) {//Malloc failed. Most-probably low on memory.break;}strlcpy(pkg_info.pkg_name,msg->pkg_name,PKG_NAME_LEN);strlcpy(tmp_pkg_name,pkg_info.pkg_name,PKG_NAME_LEN);bindApp_dur = 0;disAct_dur = 0;launching = true;time(&pkg_info.last_time_launched);compute_time_day_slot(week_day, &time_slot);QLOGI("UXEngine Updating Details: pkg_name: %s, week_day: %s, time_slot: %d %s\n", pkg_info.pkg_name, week_day, time_slot, tmp_pkg_name);update_ux_pkg_details(pkg_info, week_day, time_slot, 0);update_palm_table(msg->pkg_name, 0, 1);QLOGI("UXEngine finished ux_pkg_details update \n");free(week_day);is_in_recent_list = false;if(!enable_prefetcher){QLOGE("io prefetch is disabled");break;}//Check app is in recent listfor(ittr = 0; ittr < IOP_NO_RECENT_APP; ittr++){if(0 == strcmp(msg->pkg_name,recent_list[ittr])){is_in_recent_list = true;QLOGE("is_in_recent_list is TRUE");break;}}// IF Application is in recent list, returnif(true == is_in_recent_list){QLOGE("io prefetch is deactivate");break;}if(recent_list_cnt == IOP_NO_RECENT_APP)recent_list_cnt = 0;//Copy the package name to recent liststrlcpy(recent_list[recent_list_cnt],msg->pkg_name,PKG_LEN);recent_list_cnt++;stop_capture();stop_playback();start_playback(msg->pkg_name);}// if PID > 0 then consider as capture operationif(msg->pid > 0){if(!enable_prefetcher){QLOGE("io prefetch is disabled");break;}if(true == is_in_recent_list){QLOGE("io prefetch Capture is deactivated ");break;}stop_capture();start_capture(msg->pid,msg->pkg_name,msg->code_path,enable_prefetcher_ofr);}break;}}

pid > 0时,会先检测是否开启了iop 预取,且查看 预取的信息是否在最近的数组中,有则停止采集。接下来回收可能存在的线程,防止内存泄露,且进入start_capture中,创建一个新的线程capture_thread 进行信息采集更新至数据库中。

int start_capture(pid_t pid, char *pkg_name, char *code_path, bool ofr) {char property[PROPERTY_VALUE_MAX];capture_thread_arg *arg_bundle = NULL;int len = 0;int len_code_path = 0;if(NULL == pkg_name)return -1;len = strlen(pkg_name);if ( len <= 0){QLOGW("incorrect length for pkg_name");return -1;}arg_bundle = (capture_thread_arg *)malloc(sizeof(capture_thread_arg));if (NULL == arg_bundle)return -1;total_files = 0;QLOGE(" start_capture ENTER ");arg_bundle->pid = pid;arg_bundle->pkg_name = (char *) malloc(len+1);arg_bundle->ofr = ofr;len_code_path = strlen(code_path);if (NULL == arg_bundle->pkg_name) {free(arg_bundle);return -1;}QLOGI("%d %s %zd %d\n", pid, arg_bundle->pkg_name,strlen(pkg_name),PKG_NAME_LEN);strlcpy(arg_bundle->pkg_name, pkg_name, len+1);strlcpy(list_code_path, code_path, len_code_path+1);QLOGI("%d %s %zd %s %zd\n", pid, arg_bundle->pkg_name,strlen(pkg_name),list_code_path, strlen(code_path));halt_thread = 0;start_capture_time = (long) now_ms();time(&start_capture_time_s);property_get("vendor.iop.read_fd_interval_ms", property, "50");read_fd_interval_ms = atoi(property);if(read_fd_interval_ms <= 0) {QLOGI("read_fd_interval_ms is set to a non-positive value %d. Resetting to default value 50\n", read_fd_interval_ms);property_set("read_fd_interval_ms", "50");read_fd_interval_ms = 50;}QLOGI("property = %d\n", read_fd_interval_ms);//TBD must remove commentsif(pthread_create(&capture_pthread_id, NULL, capture_thread, arg_bundle)) {return -1;}QLOGI("start_capture EXIT");return 0;
}

在capture_thread中继续读取配置参数,根据参数是否开启某些功能,根据测试红色部分不建议开,会反响优化,使冷启动时间变慢。调用get_priv_code_files和get_priv_files获取提供的有关activity的相关文件放入数组file_list中,然后调用update_pkg_details和get_priv_files更新至数据库中。

void * capture_thread(void * arg) {int i = 0;int pkg_len = 0;int duration_counter = 0;pkg_details pkg_info;int index = 0;int halt_counter = CAPTURE_MAX_DURATION/read_fd_interval_ms;int polling_enable = 0;char property[PROPERTY_VALUE_MAX];property_get("vendor.polling_enable", property, "0");polling_enable = atoi(property);ATRACE_BEGIN("capture_thread");capture_thread_arg * arg_bundle = (capture_thread_arg *)arg;//Copy the package to same string to remove ':'pkg_len = strlen(arg_bundle->pkg_name);i = 0;while(i < pkg_len && arg_bundle->pkg_name[i] != ':'){i++;}arg_bundle->pkg_name[i] = '\0';strlcpy(list_pkg_name,arg_bundle->pkg_name,PKG_NAME_LEN);QLOGI("pkg_name  = %s",arg_bundle->pkg_name);if(polling_enable){while (duration_counter < halt_counter) {if (halt_thread) goto cleanup;QLOGI("Getting snapshot %d\n", arg_bundle->pid);get_snapshot(list_pkg_name,arg_bundle->pid);duration_counter++;usleep(read_fd_interval_ms * 1000);}}get_priv_code_files(arg_bundle->pkg_name);get_priv_files(arg_bundle);QLOGI("pkg_name = %s total_files = %d ",arg_bundle->pkg_name,total_files);// Insert package into the tablestrlcpy(pkg_info.pkg_name,arg_bundle->pkg_name,PKG_NAME_LEN);time(&pkg_info.last_time_launched);// Update Mincore dataif(arg_bundle->ofr){for(index = 0; index < total_files;index++){if(update_mincore_data(file_list[index]) != 0){file_list[index]->mincore_array = NULL;}}}update_pkg_details(pkg_info);update_file_details(arg_bundle->pkg_name, file_list, total_files);delete_mark_files();
cleanup://log a report about how many files need insert or update this timeQLOGE("# Final entry : pkg_name file_name file_time_stamp filesize file_iop_size");for(i = 0; i < total_files; i++) {QLOGE("%d. Final entry : %s %s %d %d %d\n",i,arg_bundle->pkg_name,file_list[i]->file_name, file_list[i]->file_time_stamp, file_list[i]->filesize, file_list[i]->file_iop_size);if (file_list[i]->mincore_array) {free(file_list[i]->mincore_array);file_list[i]->mincore_array = NULL;}free(file_list[i]);}ATRACE_END();free(arg_bundle->pkg_name);free(arg_bundle);QLOGI("Exit capture_thread");return NULL;
}

接下来看看get_priv_code_files 和 get_priv_files 具体获取了哪些数据?先来看看get_priv_code_files获取了哪些文件。在代码中主要获取了oat中的文件。举个例子看下:

例如.video/IqiyiVideo 目录下的 IqiyiVideo.apk lib oat等内容。capture_pkg_file是一个递归函数,主要获取文件夹里的内容,如*.odex vdex *.apk等文件

static void get_priv_code_files(char *pkg_name) {char f_name[FILE_NAME_LEN];DIR *dp;//oat firstsnprintf(f_name,FILE_NAME_LEN,"%s/oat",list_code_path);QLOGI("Dir = %s",f_name);dp = opendir(f_name);QLOGI("First dp = %p",dp);if (dp != NULL) {capture_pkg_file(pkg_name, dp, f_name, -1);closedir(dp);} else {QLOGI("dp is NULL");}//then scan apk filedp = opendir(list_code_path);QLOGI("First dp = %p",dp);if (dp != NULL) {capture_pkg_file(pkg_name, dp, list_code_path, 1);closedir(dp);} else {QLOGI("dp is NULL");}
}    

接下来看看get_priv_files获取了什么文件?
在代码中主要获取 /data/user/0/pkg_name 以及 /data/data/pkg_name 文件夹下的文件,找个文件看看如下:
第二阶段结果展望:
在这里插入图片描述

void get_priv_files(capture_thread_arg * arg_bundle)
{char f_name[FILE_NAME_LEN];DIR *dp;char trace_buf[1024];snprintf(trace_buf, sizeof(trace_buf), "Get_prive_file");ATRACE_BEGIN(trace_buf);snprintf(f_name,FILE_NAME_LEN,"/data/user/0/%s",arg_bundle->pkg_name);QLOGI("Dir = %s",f_name);dp = opendir(f_name);QLOGI("First dp = %p",dp);if (dp != NULL) {capture_pkg_file(arg_bundle->pkg_name, dp,f_name,2);closedir(dp);}else{QLOGI("dp is NULL");}snprintf(f_name,FILE_NAME_LEN,"/data/data/%s",arg_bundle->pkg_name);QLOGI("Dir = %s",f_name);dp = opendir(f_name);QLOGI("First dp = %p",dp);if (dp != NULL) {capture_pkg_file(arg_bundle->pkg_name, dp,f_name, 4);closedir(dp);}else{QLOGI("dp is NULL");}ATRACE_END();
}

接下来看看如何更新到数据库里?
先调用get_package是否已经在数据库里了,如果在给他的调用次数+1,且更新上次启动时间。如果不在则调用IO_PREFETCHER_QUERY_INSERT_PKG sql语句插入数据库中。

//Update package deails with provided attributes
int update_pkg_details(pkg_details pkg_info)
{char query_str[2048] = IO_PREFETCHER_QUERY_UPDATE_PKG_DETAILS;/* "UPDATE io_pkg_tbl  SET pkg_last_use = %lu ,pkg_use_count = pkg_use_count+1  WHERE pkg_name = '%s'"*/pkg_details temp_pkg_info;temp_pkg_info.pkg_name[0] = 0;int is_file_present  =  get_package(pkg_info.pkg_name, &temp_pkg_info);QLOGI("update_pkg_details");if(NULL == db_conn)open_db();if(is_file_present){//Updatesnprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_UPDATE_PKG_DETAILS,pkg_info.last_time_launched,pkg_info.pkg_name);QLOGI("\nQuery = %s \n",query_str);iop_query_exec(query_str);}else{//Insertsnprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_INSERT_PKG,pkg_info.pkg_name,pkg_info.last_time_launched);QLOGI("\nQuery = %s \n",query_str);iop_query_exec(query_str);}return 0;
}

这里通过get_file 获取io_pkg_file_tbl表的信息,更新到io_pkg_file_tbl表中。这里数据的更新使用了sql的事务来更新到数据库中。

// update detail for file with provided attibutes
int update_file_details(char * pkg_name,file_details *file_info[], int size)
{int i = 0;char *error=NULL;sqlite3_stmt *stmt=NULL;char trace_buf[1024];char query_str[2048] = IO_PREFETCHER_QUERY_UPDATE_FILE_DETAILS;iop_query_exec("BEGIN TRANSACTION;");for(i=0;i<size;i++) {file_details temp_file_info;temp_file_info.file_name[0]=0;int is_file_present  = get_file(pkg_name, file_info[i]->file_name, &temp_file_info);if(is_file_present) {//UpdateQLOGI("MINCORE update file %s %s study_finish %d, cache_dropped %d",pkg_name, temp_file_info.file_name,temp_file_info.study_finish, temp_file_info.cache_dropped);/** update mincore array data if this file cache is dropped and study not finished*/if (temp_file_info.cache_dropped == 1 && temp_file_info.study_finish <= 2) {snprintf(trace_buf, sizeof(trace_buf), "update file %s", file_info[i]->file_name);ATRACE_BEGIN(trace_buf);snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_UPDATE_FILE_DETAILS,file_info[i]->file_time_stamp,file_info[i]->filesize,file_info[i]->file_iop_size,pkg_name,file_info[i]->file_name);QLOGI("\nQuery = %s \n",query_str);int pages_in_range = (file_info[i]->filesize + PAGE_SIZE - 1)/PAGE_SIZE;sem_wait(&mutex);_SQLITE_CHECK_OK(sqlite3_prepare_v2(db_conn, query_str,strlen(query_str), &stmt, (const char **)&error));int index = sqlite3_bind_parameter_index(stmt, ":mincore_array");if (file_info[i]->mincore_array) {_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index,file_info[i]->mincore_array, pages_in_range, SQLITE_STATIC));}else{_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index, NULL, 0, SQLITE_STATIC));}_SQLITE_CHECK_DONE(sqlite3_step(stmt));sqlite3_reset(stmt);sqlite3_finalize(stmt);sem_post(&mutex);ATRACE_END();}else{if (file_info[i]->cache_dropped) {mark_cache_dropped(pkg_name, file_info[i]->file_name, 0);}}}else{//insert new file to dbint index;int pagesize = sysconf(_SC_PAGESIZE);QLOGI("MINCORE insert new file %s %s iopsize %d filesize %d",pkg_name, file_info[i]->file_name,file_info[i]->file_iop_size, file_info[i]->filesize);snprintf(trace_buf, sizeof(trace_buf), "insert new file %s", file_info[i]->file_name);ATRACE_BEGIN(trace_buf);snprintf(query_str,sizeof(query_str),IO_PREFETCHER_QUERY_INSER_FILE,pkg_name,file_info[i]->file_name,file_info[i]->file_time_stamp,file_info[i]->filesize,file_info[i]->file_modify_time,file_info[i]->file_iop_size);QLOGI("\nQuery = %s \n",query_str);sem_wait(&mutex);_SQLITE_CHECK_OK(sqlite3_prepare_v2(db_conn, query_str,strlen(query_str), &stmt, (const char**)&error));if (file_info[i]->mincore_array != NULL && file_info[i]->file_iop_size >pagesize) {index = sqlite3_bind_parameter_index(stmt, ":mincore_array");int pages_in_range = (file_info[i]->filesize + PAGE_SIZE - 1)/PAGE_SIZE;_SQLITE_CHECK_OK(sqlite3_bind_blob(stmt, index,file_info[i]->mincore_array, pages_in_range, SQLITE_STATIC));}_SQLITE_CHECK_DONE(sqlite3_step(stmt));sqlite3_reset(stmt);sqlite3_finalize(stmt);sem_post(&mutex);ATRACE_END();}}iop_query_exec("COMMIT TRANSACTION;");return 0;
}

此时采集信息的过程已经完成了,接下来应该是执行预取优化的过程了,可以回到iop_server 流程中的pid < 0的步骤,继续向下分析。黄色得部分主要是收集activity启动得时间信息,主要包括信息是工作日,上一次启动时间等信息,用于预测,如果预测成功权重+weight,预测失败,失败的权重+weight等信息,目前没有用到预测功能,先忽略。绿色的部分主要是看新启动的app是否是在最近启动的列表里,如果存在则不执行预取。如果都没问题则执行浅红色部分,执行获取数据库中的信息提前执行预取操作。ng)

if(msg->pid < 0){int ittr = 0;char *week_day = NULL;week_day = (char *) malloc(6*sizeof(char));// Insert package into the tableif (week_day == NULL) {//Malloc failed. Most-probably low on memory.break;}strlcpy(pkg_info.pkg_name,msg->pkg_name,PKG_NAME_LEN);strlcpy(tmp_pkg_name,pkg_info.pkg_name,PKG_NAME_LEN);bindApp_dur = 0;disAct_dur = 0;launching = true;time(&pkg_info.last_time_launched);compute_time_day_slot(week_day, &time_slot);QLOGI("UXEngine Updating Details: pkg_name: %s, week_day: %s, time_slot: %d %s\n", pkg_info.pkg_name, week_day, time_slot, tmp_pkg_name);update_ux_pkg_details(pkg_info, week_day, time_slot, 0);update_palm_table(msg->pkg_name, 0, 1);QLOGI("UXEngine finished ux_pkg_details update \n");free(week_day);is_in_recent_list = false;if(!enable_prefetcher){QLOGE("io prefetch is disabled");break;}//Check app is in recent listfor(ittr = 0; ittr < IOP_NO_RECENT_APP; ittr++){if(0 == strcmp(msg->pkg_name,recent_list[ittr])){is_in_recent_list = true;QLOGE("is_in_recent_list is TRUE");break;}}// IF Application is in recent list, returnif(true == is_in_recent_list){QLOGE("io prefetch is deactivate");break;}if(recent_list_cnt == IOP_NO_RECENT_APP)recent_list_cnt = 0;//Copy the package name to recent liststrlcpy(recent_list[recent_list_cnt],msg->pkg_name,PKG_LEN);recent_list_cnt++;stop_capture();stop_playback();start_playback(msg->pkg_name);}

Iop预取优化流程
接下来我们具体看看他是如何从数据库中获取信息,然后执行iop预取的操作的。在start_playback中开一个线程去执行。

void start_playback(char *pkg_name_arg) {char *pkg_name = NULL;int len = 0;QLOGE(" start_playback-1 ");if(NULL == pkg_name_arg)return;len = strlen(pkg_name_arg);if(len <= 0){QLOGW("Incorrect length for pkg_name_arg");return;}pkg_name = (char *) malloc(len + 1);if (NULL == pkg_name)return;strlcpy(pkg_name, pkg_name_arg,len+1);QLOGI(" %s %zd\n", pkg_name,strlen(pkg_name));if(pthread_create(&start_playback_pthread_id, NULL, start_playback_thread, pkg_name)) {return ;}QLOGI("Exit playback");return ;
}

接下来是Iop的核心操作。get_total_file(pkg_name)这个操作是检查数据库中是否有数据,如果没有,则没有预取操作了。如果num_file>0,则表示有数据,则调用get_file_list,获取pkg_name 对应的文件信息存储到file_detail_ptr中,接下来的一对操作主要是进行数据的标记,是否要清理缓存,根据数据标签把一些不确定的因素去掉。此时已经获取到了预取信息到file_detail_ptr,调用iop_fileplayback_run(file_detail_ptr)进行预取,最后调用 commit_filelist_info ()将更新的信息更新到数据库里。

static void* start_playback_thread(void *pkg_name_arg)
{file_details *file_detail_ptr;int num_file = 0;int num_of_files = 0;char *pkg_name = (char *)pkg_name_arg;int i = 0;total_data = 0;struct stat file_stat;int pagesize = sysconf(_SC_PAGESIZE);ATRACE_BEGIN("start_playback_thread: Enter");QLOGI("pkg_name = %s",pkg_name);num_file = get_total_file(pkg_name);if( num_file <= 0){QLOGI("no file to read get_total_file %d", num_file);remove_pkg(pkg_name);free(pkg_name);ATRACE_END();return NULL;}file_detail_ptr = (file_details *) malloc(sizeof(file_details) * num_file);if (NULL == file_detail_ptr){QLOGI("Fail to allocate memory");remove_pkg(pkg_name);free(pkg_name);ATRACE_END();return NULL;}num_of_files = get_file_list(pkg_name,file_detail_ptr,num_file);QLOGI("num_of_files = %d",num_of_files);if (num_of_files <= 0){QLOGI("no file to read get_file_list");remove_pkg(pkg_name);free(file_detail_ptr);free(pkg_name);ATRACE_END();return NULL;}if( num_file > MAX_NUM_FILE){//free mincore array that exceed MAX_NUM_FILEfor (i=MAX_NUM_FILE; i<num_of_files; i++) {if (file_detail_ptr[i].mincore_array){free(file_detail_ptr[i].mincore_array);file_detail_ptr[i].mincore_array=NULL;}}num_file = MAX_NUM_FILE;}for(i = 0; i < num_file;i++){int fd = -1;char trace_buf[4096];if (file_detail_ptr[i].disabled) {QLOGE("skip preload disabled file %d %s", i, file_detail_ptr[i].file_name);continue;}if (file_detail_ptr[i].study_finish && file_detail_ptr[i].file_iop_size < pagesize) {file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_NOT_DROPPED;QLOGE("skip preload ZERO needed file %d %s", i, file_detail_ptr[i].file_name);continue;}fd = open(file_detail_ptr[i].file_name, O_RDONLY);if(fd == -1){mark_for_delete(pkg_name,file_detail_ptr[i].file_name);continue;}if ( fstat( fd, &file_stat ) < 0 ){QLOGI("fail to get file stat");mark_for_delete(pkg_name,file_detail_ptr[i].file_name);close(fd);continue;// goto  next file}if ( file_stat.st_size == 0 ){// File zie is zeromark_for_delete(pkg_name,file_detail_ptr[i].file_name);close(fd);continue;// goto  next file}snprintf(trace_buf,4096,"Opening file %s size = %d",file_detail_ptr[i].file_name, (int)file_stat.st_size);ATRACE_BEGIN(trace_buf);if (file_stat.st_mtime != file_detail_ptr[i].file_modify_time) {if (strstr(file_detail_ptr[i].file_name,".apk")!=NULL&& (strstr(file_detail_ptr[i].file_name,"/data/app/")!=NULL ||strstr(file_detail_ptr[i].file_name, "/system/app/")!=NULL ||strstr(file_detail_ptr[i].file_name, "/system/priv-app/")!=NULL)){//readonly /data/app/base.apkQLOGI("apk file %s modify time changed %d/%ld, \remove files belong to pkg, and restudy",file_detail_ptr[i].file_name,file_detail_ptr[i].file_modify_time, file_stat.st_mtime);snprintf(trace_buf, sizeof(trace_buf), "apk file %s upgrade, remove pkg", file_detail_ptr[i].file_name);ATRACE_BEGIN(trace_buf);file_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_DELETE_PKG;close(fd);ATRACE_END();ATRACE_END(); // end for opening fileATRACE_END(); //end for start_playback_thread enterreturn NULL;} else {QLOGI("file %s modify time changed update db", file_detail_ptr[i].file_name);//mark study_finish=0 and cache_dropped=1snprintf(trace_buf, sizeof(trace_buf), "file %s modifie mark update", file_detail_ptr[i].file_name);ATRACE_BEGIN(trace_buf);//restudy if file modify time changed for .odex and .dex fileif (strstr(file_detail_ptr[i].file_name, ".odex") != NULL|| strstr(file_detail_ptr[i].file_name, ".dex") != NULL|| strstr(file_detail_ptr[i].file_name, ".vdex") != NULL){file_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_UPDATE;file_detail_ptr[i].file_modify_time = file_stat.st_mtime;file_detail_ptr[i].filesize = file_stat.st_size;drop_file_cache(fd, &(file_detail_ptr[i]));} else {//disable unstable filefile_detail_ptr[i].mark_for_delete = NEED_MARK_FOR_DISABLE;file_detail_ptr[i].disabled = 1;}ATRACE_END();}close(fd);continue;}//try if need drop cache or playbackif (file_detail_ptr[i].study_finish == 0) {QLOGI("Drop cache of file %s", file_detail_ptr[i].file_name);snprintf(trace_buf, sizeof(trace_buf), "Drop cache of file %s",file_detail_ptr[i].file_name);ATRACE_BEGIN(trace_buf);drop_file_cache(fd, &(file_detail_ptr[i]));file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_DROPPED;ATRACE_END();} else {QLOGI("io preload of file %s", file_detail_ptr[i].file_name);snprintf(trace_buf, sizeof(trace_buf), "add file %s %.2f / %.2f MB to task",file_detail_ptr[i].file_name,file_detail_ptr[i].file_iop_size/1024.0/1024,file_detail_ptr[i].filesize/1024.0/1024);ATRACE_BEGIN(trace_buf);iop_fileplayback_run(file_detail_ptr[i],file_stat,fd);ATRACE_END();if (file_detail_ptr[i].cache_dropped) {file_detail_ptr[i].cache_dropped = NEED_MARK_CACHE_NOT_DROPPED;}}close(fd);if(total_data > MAX_TOTAL_DATA){QLOGI("Max total size limit reached Total = %d",total_data);//STOP reading and max total reachedbreak;}ATRACE_END(); // end for opening file}commit_filelist_info(file_detail_ptr, num_file);delete_mark_files();ATRACE_END(); //end for start_playback_thread enterQLOGI(" Total Data = %.2f MB", total_data/1024.0/1024);for (i=0; i<num_of_files; i++) {if (file_detail_ptr[i].mincore_array) {free(file_detail_ptr[i].mincore_array);file_detail_ptr[i].mincore_array=NULL;}}free(pkg_name);free(file_detail_ptr);return NULL;
}

下面来看看iop_fileplayback_run具体执行了什么操作?
在这里主要是通过posix_fadvise 系统调用 + POSIX_FADV_WILLNEED 参数执行了预读操作,将需要加载的文件提前读到了pagecache里,使启动时读取该部分文件能直接命中缓存加速启动速度。具体可以参照posix_fadvise的作用 linux系统函数posix_fadvise_九品神元师的博客-CSDN博客。

/* Run playback operation */
static void *iop_fileplayback_run(file_details info, struct stat file_stat, int fd)
{int page_itrr;int total_mincore_pages = (info.file_iop_size+PAGE_SIZE-1)/PAGE_SIZE;int total_pages = (file_stat.st_size+PAGE_SIZE-1)/PAGE_SIZE;char *mincore_array = info.mincore_array;char trace_buf[1024];if (total_mincore_pages > 0) {snprintf(trace_buf, sizeof(trace_buf), "%s read size:%.4fMB",info.file_name,total_mincore_pages*4/1024.0);}else{snprintf(trace_buf, sizeof(trace_buf), "%s read size:%.4fMB",info.file_name,file_stat.st_size/1024.0/1024);}ATRACE_BEGIN(trace_buf);int count_pages=0;int read_data = 0;int ret = 0;QLOGI("MINCORE, PLAYBACK %s START, mincore_array: %p",info.file_name, mincore_array);//load each page by posix_fadvise here to avoid read_ahead//and decrease CPU usage of copy io content result to local viriableif (mincore_array == NULL) {//if all file need read to cacheint len_size = file_stat.st_size;ret = posix_fadvise(fd, 0, len_size, POSIX_FADV_WILLNEED);read_data += len_size;if (ret !=0) {QLOGE("posix_fadvise failed: %s", strerror(errno));}} else {//load file content based on mincore resultcount_pages=0;for(page_itrr = 0; page_itrr < total_pages; page_itrr++) {if (mincore_array[page_itrr] & 0x1) {if (count_pages>=total_mincore_pages)break;ret = posix_fadvise(fd, page_itrr*PAGE_SIZE, PAGE_SIZE, POSIX_FADV_WILLNEED);if (ret !=0) {QLOGE("posix_fadvise failed: %s", strerror(errno));}read_data += PAGE_SIZE;count_pages++;}}}//after finish preload, mark file cache advise as normalret = posix_fadvise(fd, 0, file_stat.st_size, POSIX_FADV_NORMAL);if (ret !=0) {QLOGE("posix_fadvise failed: %s", strerror(errno));}close(fd);total_data += read_data;QLOGI("MINCORE, PLAYBACK %s END real read file size = %.4fMB",info.file_name, read_data/1024.0/1024);ATRACE_END();return NULL;
}

学前疑问解答

  1. Iop是如何怎么预取的?
    主要通过posix_fadvise 这个系统调用配合相应的参数将所需要的文件预取到PageCache里。
  2. 数据库的表是如何存放数据的?
    数据库中的数据主要是从codepath,和 遍历pid 中的fd所指向的链接的信息进行筛选存入数据库的表中,iop用到的表主要存app的.apk,vdex,.so等文件,并记录被使用的时间,和使用到的次数。
预期日期 截止日期 结果
2022/11/14 2022/11/25 理解Iop底层机制,输出一篇关于Iop底层原理的文档
第三阶段

第三阶段属于测试阶段,在这个阶段,我应该已经成功启动了Iop机制,且已经掌握了90%的Iop原理,可以比较灵活的使用BoostFrame了。这阶段的主要目标是测试在Iop开启的条件下,App冷启动和热启动时间的效率比,当然测试的话还是使用自动化脚本比较方便,第三阶段的主要目标是输出一个相关脚本,并输出一篇性能分析文档。

第三阶段结果展望:

预期日期 截止日期 结果
2022/11/24 2022/12/5 输出测试文档

集成之后结果不如人意,从原理上讲应该可以优化,但集成之后并没有优化。心累…

代码编程
赞赏

相关文章

她是个好女孩
初一有感…
爱你……不愿放弃
爱自己
等你
对不起,我爱你