summaryrefslogtreecommitdiff
path: root/kernel (follow)
Commit message (Collapse)AuthorAge
...
* | | sched: window-stats: Fix accounting bug in legacy modeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | TASK_UPDATE event currently does not result in increment of rq->curr_runnable_sum in legacy mode, which is wrong. As a result, cpu busy time reported under legacy mode could be incorrect. Change-Id: Ifa76c735a0ead23062c1a64faf97e7b801b66bf9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Note legacy mode in fork() and exit()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In legacy mode, mark_task_starting() should avoid adding (new) task's (initial) demand to rq->curr_runnable_sum and rq->prev_runnable_sum. Similarly exit() should avoid removing (exiting) task's demand from rq->curr_runnable_sum and rq->prev_runnable_sum (as those counters don't include task's demand and partial_demand values in legacy mode). Change-Id: I26820b1ac5885a9d681d363ec53d6866a2ea2e6f Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Fix reference to stale task_struct in try_to_wake_up()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | try_to_wake_up() currently drops p->pi_lock and later checks for need to notify cpufreq governor on task migrations or wakeups. However the woken task could exit between the time p->pi_lock is released and the time the test for notification is run. As a result, the test for notification could refer to an exited task. task_notify_on_migrate(p) could thus lead to invalid memory reference. Fix this by running the test for notification with task's pi_lock held. Change-Id: I1c7a337473d2d8e79342a015a179174ce00702e1 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Remove hack to enable/disable HMP scheduling extensionsSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current method of turning HMP scheduling extensions on or off based on the number of CPUs is inappropriate as there may be SoCs with 4 or less cores that require the use of these extensions. Remove this hack as HMP extensions will now be enabled/disabled via command line options. Change-Id: Id44b53c2c3b3c3b83e1911a834e2c824f3958135 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: fix wrong load_scale_factor/capacity/nr_big/small_tasksSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A couple bugs exist with incorrect use of cpu_online_mask in pre/post_big_small_task() functions, leading to potentially incorrect computation of load_scale_factor/capacity/nr_big/small_tasks. pre/post_big_small_task_count_change() use cpu_online_mask in an unreliable manner. While local_irq_disable() in pre_big_small_task_count_change() ensures a cpu won't go away in cpu_online_mask, nothing prevents a cpu from coming online concurrently. As a result, cpu_online_mask used in pre_big_small_task_count_change() can be inconsistent with that used in post_big_small_task_count_change() which can lead to an attempt to unlock rq->lock which was not taken before. Secondly, when either max_possible_freq or min_max_freq is changing, it needs to trigger recomputation of load_scale_factor and capacity for *all* cpus, even if some are offline. Otherwise, an offline cpu could later come online with incorrect load_scale_factor/capacity. While it should be sufficient to scan online cpus for updating their nr_big/small_tasks in post_big_small_task_count_change(), unfortunately it sounds pretty hard to provide a stable cpu_online_mask when its called from cpufreq_notifier_policy(). cpufreq framework can trigger a CPUFREQ_NOTIFY notification in multiple contexts, some in cpu-hotplug paths, which makes it pretty hard to guess whether get_online_cpus() can be taken without causing deadlocks or not. To workaround the insufficient information we have about the hotplug-safety context when CPUFREQ_NOTIFY is issued, have post_big_small_task_count_change() traverse all possible cpus in updating nr_big/small_task_count. CRs-Fixed: 717134 Change-Id: Ife8f3f7cdfd77d5a21eee63627d7a3465930aed5 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: add check for cpu idleness when using C-state informationSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Task enqueue on a CPU occurs prior to that CPU exiting an idle state. For the time duration between enqueue and idle exit, the CPU C-state information can no longer be relied on for further task placement since already enqueued/waiting tasks are not taken into account. The small task placement algorithm implicitly assumes a non zero C-state implies an idle CPU. Since this assumption is incorrect for the duration described above, make the cpu_idle() check explicit. This problem can lead to task packing beyond the mostly_idle threshold. Change-Id: Idb5be85705d6b15f187d011ea2196e1bfe31dbf2 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: extend sched_task_load tracepoint to indicate small tasksSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While debugging its always useful to know whether a task is small or not to determine the scheduling algorithm being used. Have the sched_task_load tracepoint indicate this information rather than having to do manual calculations for every task placement. Change-Id: Ibf390095f05c7da80df1ebfe00f4c5af66c97d12 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Add C-state tracking to the sched_cpu_load trace eventSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | C-state information is used by the scheduler for small task placement decisions. Track this information in the sched_cpu_load trace event. Also add the trace event in best_small_task_cpu(). This will help better understand small task placement decisions. Change-Id: Ife5f05bba59f85c968fab999bd13b9fb6b1c184e Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: window-stats: add a new AVG policySyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current WINDOW_STATS_AVG policy is actually a misnomer since it uses the maximum value of the runtime in the recent window and the average of the past ravg_hist_size windows. Add a policy that only uses the average and call it WINDOW_STATS_AVG policy. Rename all the other polices to make them shorter and unambiguous. Change-Id: I080a4ea072a84a88858ca9da59a4151dfbdbe62c Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Fix compile errorSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sched_get_busy(), sched_set_io_is_busy() and sched_set_window() need to be defined only when CONFIG_SCHED_FREQ_INPUT is defined, otherwise we get compilation error related to dual definition of those routines Change-Id: Ifd5c9b6675b78d04c2f7ef0e24efeae70f7ce19b Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched.h] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: update ld_moved for active balance from the load balancerSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ld_moved is currently left set to 0 when the load balancer calls upon active balance. This behavior is incorrect as it prevents the termination of load balance for parent sched domains. Currently the feature is used quite frequently for power active balance and sched boost. This means that while sched boost is in effect we could run into a scenario where a more power efficient newly idle big CPU first triggers active migration from a less power efficient busy big CPU. It then continues to load balance at the cluster level causing active migration for a task running on a little CPU. Consequently the more power efficient big CPU ends up with two tasks where as the less power efficient big CPU may become idle. Fix this problem by updating ld_moved when active migration has been requested. Change-Id: I52e84eafb77249fd9378ebe531abe2d694178537 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: actively migrate tasks to idle big CPUs during sched boostSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sched boost feature is currently tick driven, i.e. task placement decisions only take place at a tick (or wakeup). The load balancer does not have any knowledge of boost being in effect. Tasks that are woken up on a little CPU when all big CPUs are busy will continue executing there at least until the next tick even if one of the big CPUs becomes idle. Reduce this latency by adding support for detecting whether boost is in effect or not in the load balancer. If boost is in effect any big CPU running idle balance will trigger active migration from a little CPU with the highest task load. Change-Id: Ib2828809efa0f9857f5009b29931f63b276a59f3 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: always do idle balance with a NEWLY_IDLE idle environmentSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With the introduction of energy aware scheduling, if idle_balance() is to be called on behalf of a different CPU which is idle, CPU_IDLE is used in the environment for load_balance(). This, however, introduces subtle differences in load calculations and policies in the load balancer. For example there are restrictions on which CPU is permitted to do load balancing during !CPU_NEWLY_IDLE (see update_sg_lb_stats) and find_busiest_group() uses different criteria to detect the presence of a busy group. There are other differences as well. Revert back to using the NEWLY_IDLE environment irrespective of whether idle_balance() is called for the newly idle CPU or on behalf on already existing idle CPU. This will ensure that task movement logic while doing idle balance remains unaffected. Change-Id: I388b0ad9a38ca550667895c8ed19628f3d25ce1a Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: fix bail condition in bail_inter_cluster_balance()Syed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Following commit efcad25cbfb (revert "sched: influence cpu_power based on max_freq and efficiency), all CPUs in the system have the same cpu_power and consequently the same group capacity. Therefore, the check in bail_inter_cluster_balance() can now no longer be used to distinguish a higher performance cluster from one with lower performance. The check is currently broken and always returns true for every load balancing attempt. Fix this by using runqueue capacity instead which can still be used as a good measure of cluster capabilities. Change-Id: Idecfd1ed221d27d4324b20539e5224a92bf8b751 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Initialize env->loop variable to 0Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | load_balance() function does not explicitly initialize env->loop variable to 0. As a result, there is a vague possibility of move_tasks() hitting a very long (unnecessary) loop when its unable to move tasks from src_cpu. This can lead to unpleasant results like a watchdog bark. Fix this by explicitly initializing env->loop variable to 0 (in both load_balance() and active_load_balance_cpu_stop()). Change-Id: I36b84c91a9753870fa16ef9c9339db7b706527be Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: use policy_mutex in sched_set_window()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Several configuration variable change will result in reset_all_window_stats() being called. All of them, except sched_set_window(), are serialized via policy_mutex. Take policy_mutex in sched_set_window() as well to serialize use of reset_all_window_stats() function Change-Id: Iada7ff8ac85caa1517e2adcf6394c5b050e3968a Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Avoid taking all cpu's rq->lock for longSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | reset_all_window_stats() walks task-list with all cpu's rq->lock held, which can cause spinlock timeouts if task-list is huge (and hence lead to a spinlock bug report). Avoid this by walking task-list without cpu's rq->lock held. Change-Id: Id09afd8b730fa32c76cd3bff5da7c0cd7aeb8dfb Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window_stats: Add "disable" mode supportSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "disabled" mode (sched_disble_window_stats = 1) disables all window-stats related activity. This is useful when changing key configuration variables associated with window-stats feature (like policy or window size). Change-Id: I9e55c9eb7f7e3b1b646079c3aa338db6259a9cfe Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Fix exit raceSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Exiting tasks are removed from tasklist and hence at some point will become invisible to do_each_thread/for_each_thread task iterators. This breaks the functionality of reset_all_windows_stats() which *has* to reset stats for *all* tasks. This patch causes exiting tasks stats to be reset *before* they are removed from tasklist. DONT_ACCOUNT bit in exiting task's ravg.flags is also marked so that their remaining execution time is not accounted in cpu busy time counters (rq->curr/prev_runnable_sum). reset_all_windows_stats() is thus guaranteed to return with all task's stats reset to 0. Change-Id: I5f101156a4f958c1b3f31eb0db8cd06e621b75e9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Provide a wrapper function to reset task's window statistics. This will be reused by a subsequent patch Change-Id: Ied7d32325854088c91285d8fee55d5a5e8a954b3 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: legacy modeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Support legacy mode, which results in busy time being seen by governor that is close to what it would have seen via existing APIs i.e get_cpu_idle_time_us(), get_cpu_iowait_time_us() and get_cpu_idle_time_jiffy(). In particular, legacy mode means that only task execution time is counted in rq->curr_runnable_sum and rq->prev_runnable_sum. Also task migration does not result in adjustment of those counters. Change-Id: If374ccc084aa73f77374b6b3ab4cd0a4ca7b8c90 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Collapse duplicated comments about keeping few of sysctl knobs initialized to same value as their non-sysctl copies Change-Id: Idc8261d86b9f36e5f2f2ab845213bae268ae9028 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Remove code duplication associated with update of various window-stats related sysctl tunables Change-Id: I64e29ac065172464ba371a03758937999c42a71f Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | add_task_demand() and 'long_sleep' calculation in it are not strictly required. rq_freq_margin() check for need to change frequency, which removes need for long_sleep calculation. Once that is removed, need for add_task_demand() vanishes. Change-Id: I936540c06072eb8238fc18754aba88789ee3c9f5 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minior conflict in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: Remove unused prev_window variableSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | Remove unused prev_window variable in 'struct ravg' Change-Id: I22ec040bae6fa5810f9f8771aa1cb873a2183746 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: disable frequency notifications by defaultSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The frequency notifications from the scheduler do not currently respect synchronous topologies. If demand on CPU 0 is driving frequency high and CPU 1 is in the same frequency domain, and demand on CPU 1 is low, frequency notifiers will be continuously sent by CPU 1 in an attempt to have its frequency lowered. Until the notifiers are fixed, disable them by default. They can still be re-enabled at runtime. Change-Id: Ic8a927af2236d8fe83b4f4a633b20a8ddcfba359 Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: fix misalignment between requested and actual windowsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When set_window_start() is first executed sched_clock() has not yet stabilized. Refresh the sched_init_jiffy and sched_clock_at_init_jiffy values until it is known that sched_clock has stabilized - this will be the case by the time a client calls the sched_set_window() API. Change-Id: Icd057707ff44c3b240e5e7e96891b23c95733daa Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: Make RAVG_HIST_SIZE tunableOlav Haugan2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make RAVG_HIST_SIZE available from /proc/sys/kernel/sched_ravg_hist_size to allow tuning of the size of the history that is used in computation of task demand. CRs-fixed: 706138 Change-Id: Id54c1e4b6e974a62d787070a0af1b4e8ce3b4be6 Signed-off-by: Olav Haugan <ohaugan@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in sysctl.h] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Fix possibility of "stuck" reserved flagSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | check_for_migration() could mark a thread for migration (in rq->push_task) and invoke active_load_balance_cpu_stop(). However that thread could get migrated to another cpu by the time active_load_balance_cpu_stop() runs, which could fail to clear reserved flag for a cpu and drop task_sruct reference when cpu has only one task (stopper thread running active_load_balance_cpu_stop()). This would cause a cpu to have reserved bit stuck, which prevents it from being used effectively. Fix this by having active_load_balance_cpu_stop() drop reserved bit always. Change-Id: I2464a46b4ddb52376a95518bcc95dd9768e891f9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: initialize env->flags variable to 0Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | env->flags and env->new_dst_cpu fields are not initialized in load_balance() function. As a result, load_balance() could wrongly see LBF_SOME_PINNED flag set and access (bogus) new_dst_cpu's runqueue leading to invalid memory reference. Fix this by initializing env->flags field to 0. While we are at it, fix similar issue in active_load_balance_cpu_stop() function, although there is no harm present currently in that function with uninitialized env->flags variable. Change-Id: Ied470b0abd65bf2ecfa33fa991ba554a5393f649 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: 64-bit type for curr/prev_runnable_sumSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Expand rq->curr_runnable_sum and rq->prev_runnable_sum to be 64-bit counters as otherwise they can easily overflow when a cpu has many tasks. Change-Id: I68ab2658ac6a3174ddb395888ecd6bf70ca70473 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Allow acct_wait_time to be tunedSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | Add sysctl interface to tune sched_acct_wait_time variable at runtime Change-Id: I38339cdb388a507019e429709a7c28e80b5b3585 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Account interrupt handling time as busy timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Account cycles spent by idle cpu handling interrupts (irq or softirq) towards its busy time. Change-Id: I84cc084ced67502e1cfa7037594f29ed2305b2b1 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: Account idle time as busy timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Provide a knob to consider idle time as busy time, when cpu becomes idle as a result of io_schedule() call. This will let governor parameter 'io_is_busy' to be appropriately honored. Change-Id: Id9fb4fe448e8e4909696aa8a3be5a165ad7529d3 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Account wait timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Extend window-based task load accounting mechanism to include wait-time as part of task demand. A subsequent patch will make this feature configurable at runtime. Change-Id: I8e79337c30a19921d5c5527a79ac0133b385f8a9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: update task demand on tickSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A task can execute on a cpu for a long time without being preempted or migrated. In such case, its demand can become outdated for a long time. Prevent that from happening by updating demand of currently running task during scheduler tick. Change-Id: I321917b4590635c0a612560e3a1baf1e6921e792 CRs-Fixed: 698662 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial merge conflict in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Fix herding issueSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | check_for_migration() could run concurrently on multiple cpus, resulting in multiple tasks wanting to migrate to same cpu. This could cause cpus to be underutilized and lead to increased scheduling latencies for tasks. Fix this by serializing select_best_cpu() calls from cpus running check_for_migration() check and marking selected cpus as reserved, so that subsequent call to select_best_cpu() from check_for_migration() will skip reserved cpus. Change-Id: I73a22cacab32dee3c14267a98b700f572aa3900c Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: window-stats: print window size in /proc/sched_debugSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Printing window size in /proc/sched_debug would provide useful information to debug scheduler issues. Change-Id: Ia12ab2cb544f41a61c8a1d87bf821b85a19e09fd Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Extend ftrace event to record boost and reason codeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new ftrace event to record changes to boost setting. Also extend sched_task_load() ftrace event to record boost setting and reason code passed to select_best_cpu(). This will be useful for debug purpose. Change-Id: Idac72f86d954472abe9f88a8db184343b7730287 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Avoid needless migrationSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Restrict check_for_migration() to operate on fair_sched class tasks only. Also check_for_migration() can result in a call to select_best_cpu() to look for a better cpu for currently running task on a cpu. However select_best_cpu() can end up suggesting a cpu that is not necessarily better than the cpu on which task is running currently. This will result in unnecessary migration. Prevent that from happening. Change-Id: I391cdda0d7285671d5f79aa2da12eaaa6cae42d7 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Drop active balance request upon cpu going offlineSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A cpu could mark its currently running task to be migrated to another cpu (via rq->push_task/rq->push_cpu) and could go offline before active load balance handles the request. In such case, clear the active load balance request. Change-Id: Ia3e668e34edbeb91d8559c1abb4cbffa25b1830b Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: trigger immediate migration of tasks upon boostSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently turning on boost does not immediately trigger migration of tasks from lower capacity cpus. Tasks could incur migration latency of up to one timer tick (when check_for_migration() is run). Fix this by triggering a migration check on cpus with lower capacity as soon as boost is turned on for first time. Change-Id: I244649f9cb6608862d87631325967b887b7f4b7e Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: Extend boost benefit for small and low-prio tasksSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Allow small and low-prio tasks to benefit from boost, which is expected to last for a short duration. Any task that wishes to run during that short period is allowed boost benefit. Change-Id: I02979a0c5feeba0f1256b7ee3d73f6b283fcfafa Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Handle policy change properlySrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sched_window_stat_policy influences task demand and thus various statistics maintained per-cpu like curr_runnable_sum. Changing policy non-atomically would lead to improper accounting. For example, when task is enqueued on a cpu's runqueue, its demand that is added to rq->cumulative_runnable_avg could be based on AVG policy and when its dequeued its demand that is removed can be based on MAX, leading to erroneous accounting. This change causes policy change to be "atomic" i.e all cpu's rq->lock are held and all task's window-stats are reset before policy is changed. Change-Id: I6a3e4fb7bc299dfc5c367693b5717a1ef518c32d CRs-Fixed: 687409 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched/sysctl.h. Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: Reset all window statsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, few of the window statistics for tasks are not reset when window size is changing. Fix this to completely reset all window statistics for tasks and cpus. Move the reset code to a function, which can be reused by a subsequent patch that resets same statistics upon policy change. Change-Id: Ic626260245b89007c4d70b9a07ebd577e217f283 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Additional error checking in sched_set_window()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Check for invalid window size passed as argument to sched_set_window() Also move up local_irq_disable() call to avoid thread from being preempted during calculation of window_start and its comparison against sched_clock(). Use right macro to evluate whether window_start argument is ahead in time or not. Change-Id: Idc0d3ab17ede08471ae63b72a2d55e7f84868fd6 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Fix incorrect calculation of partial_demandSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | When using MAX_POLICY, partial_demand is calculated incorrectly as 0. Fix this by picking maximum of previous 4 windows and most recent sample. Change-Id: I27850a510746a63b5382c84761920fc021b876c5 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Fix potential wrong use of rqSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 'rq' reference to a cpu where a waking task last ran can be potentially incorrect leading to incorrect accounting. This happens when task_cpu() changes between points A & B in try_to_wake_up() listed below: try_to_wake_up() { cpu = src_cpu = task_cpu(p); rq = cpu_rq(src_cpu); -> Point A .. while (p->on_cpu) cpu_relax(); smp_rmb(); raw_spin_lock(&rq->lock); -> Point B Fix this by initializing 'rq' variable after task has slept (its on_cpu field becomes 0). Also avoid adding task demand to its old cpu runqueue (prev_runnable_sum) in case it's gone offline. Change-Id: I9e5d3beeca01796d944137b5416805b983a6e06e Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: set initial task load to just above a small taskSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To maximize power savings, set the intial load of newly created tasks to just above a small task. Setting it below the small task threshold would cause new tasks to be packed which is very likely too aggressive. Change-Id: Idace26cc0252e31a5472c73534d2f5277a1e3fa4 Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched/fair: Check whether any CPUs are availableOlav Haugan2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a possibility that there are no allowed CPUs online when we try to select the best cpu for a small task. Add a check to ensure we don't continue if there are no CPUs available. CRs-fixed: 692505 Change-Id: Iff955fb0d0b07e758a893539f7bc8ea8aa09d9c4 Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-14 02:09:11 +0000