summaryrefslogtreecommitdiff
path: root/kernel (follow)
Commit message (Collapse)AuthorAge
...
* | | sched: enable hmp, power aware scheduling for targets with > 4 CPUsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enabling and disabling hmp/power-aware scheduling is meant to be done via kernel command line options. Until that is fully supported however, take advantage of the fact that current targets with more than 4 CPUs will need these features. Change-Id: I4916805881d58eeb54747e4b972816ffc96caae7 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: remove sysctl control for HMP and power-aware task placementSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no real need to control HMP and power-aware task placement at runtime after kernel has booted. Boot-time control should be sufficient. Not allowing for runtime (sysctl) support simplifies the code quite a bit. Also rename sysctl_sched_enable_hmp_task_placement to be shorter. Change-Id: I60cae51a173c6f73b79cbf90c50ddd41a27604aa Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict. p->nr_cpus_allowed == 1 has moved to core.c Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: support legacy mode betterSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It should be possible to bypass all HMP scheduler changes at runtime by setting sysctl_sched_enable_hmp_task_placement and sysctl_sched_enable_power_aware to 0. Fix various code paths to honor this requirement. Change-Id: I74254e68582b3f9f1b84661baf7dae14f981c025 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed conflict in rt.c, p->nr_cpus_allowed == 1 is now moved in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Avoid the long if() block of code in set_task_cpu(). Move that code to its own function Change-Id: Ia80a99867ff9c23a614635e366777759abaccee4 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Add BUG_ON when task_cpu() is incorrectSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It would be fatal if task_cpu() information for a task does not accurately represent the cpu on which its running. All sorts of wierd issues can arise if that were to happen! Add a BUG_ON() in context switch to detect such cases. Change-Id: I4eb2c96c850e2247e22f773bbb6eedb8ccafa49c Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: avoid active migration of tasks not in TASK_RUNNING stateSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Avoid wasting effort in migrating tasks that are about to sleep. Change-Id: Icf9520b1c8fa48d3e071cb9fa1c5526b3b36ff16 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: fix up task load during migrationSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix the hack to set task's on_rq to 0 during task migration. Task's load is temporarily added back to its runqueue so that update_task_ravg() can fixup task's load when its demand is changing. Task's load is removed immediately afterwards. Temporarily setting p->on_rq to 0 introduces a race condition with try_to_wake_up(). Another task (task A) may be attempting to wake up the migrating task (task B). As long as task A sees task B's p->on_rq as 1, the wake up will not continue. Changing p->on_rq to 0, then back to 1, allows task A to continue "waking" task B, at which point we have both try_to_wake_up and the migration code attempting to set the cpu of task B at the same time. CRs-Fixed: 695071 Change-Id: I525745f144da4ffeba1d539890b4d46720ec3ef1 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: avoid pushing tasks to an offline CPUPrasad Sodagudi2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently active_load_balance_cpu_stop is run by cpu stopper and it pushes running tasks off the busiest CPU onto idle target CPU. But there is no check to see whether target cpu is offline or not before pushing the tasks. With the introduction of active migration in the scheduler tick path (see check_for_migration()) there have been instances of attempts to migrate tasks to offline CPUs. Add a check as to whether the target cpu is online or not to prevent scheduling on offline CPUs. Change-Id: Ib8ac7f8aeabd3ca7365f3eae977075952dab4f21 Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: Add a per rq max_possible_capacity for use in power calculationsSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the absence of a power driver providing real power values, the scheduler currently defaults to using capacity of a CPU as a measure of power. This, however, is not a good measure since the capacity of a CPU can change due to thermal conditions and/or other hardware restrictions. These frequency restrictions have no effect on the power efficiency of those CPUs. Introduce max possible capacity of a CPU to track an absolute measure of capacity which translates into a good absolute measure of power efficiency. Max possible capacity takes the max possible frequency of CPUs into account instead of max frequency. Change-Id: Ia970b853e43a90eb8cc6fd990b5c47fca7e50db8 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Disable interrupts when holding the rq lock in sched_get_busy()Syed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Interrupts can end up waking processes on the same cpu as the one for which sched_get_busy() is called. Since sched_get_busy() takes the rq lock this can result in a deadlock as the same rq lock is required to enqueue the waking up task. Fix the deadlock by disabling interrupts when taking the rq lock. Change-Id: I46e14a14789c2fb0ead42363fbaaa0a303a5818f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Make wallclock more accurateSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | update_task_ravg() in context switch uses wallclock that is updated before running put_prev_task() and pick_next_task(), both of which can take some time. Its better to update wallclock after those routines, leading to more accurate accounting. Change-Id: I882b1f0e8eddd2cc17d42ca2ab8f7a2841b8d89a Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Make task and CPU load calculations safe from truncationSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Load calculations have been modified to accept and return 64 bit values. Fix up all the places where we make such calculations to store the result in 64 bit variables. This is necessary to avoid issues caused by truncation of values. While at it update scale_task_load() to scale_load_to_cpu(). This is because the API is used to scale load of both individual tasks as well as the cumulative load of CPUs. In this sense the name was a misnomer. Also clean up power_cost() to use max_task_load(). Change-Id: I51e683e1592a5ea3c4e4b2b06d7a7339a49cce9c Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed conflict in power_cost(). power_cost() now supports sched_use_pelt=1 back again. Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched/fair: Introduce C-state aware task placement for small tasksSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Small tasks execute for small durations. This means that the power cost of taking CPUs out of a low power mode outweigh any performance advantage of using an idle core or power advantage of using the most power efficient CPU. Introduce C-state aware task placement for small tasks. This requires a two pass approach where we first determine the most power effecient CPU and establish a band of CPUs offering a similar power cost for the task. The order of preference then is as follows: 1) Any mostly idle CPU in active C-state in the same power band. 2) A CPU with the shallowest C-state in the same power band. 3) A CPU with the least load in the same power band. 4) Lowest power CPU in a higher power band. The patch also modifies the definition of a small task. Small tasks are now determined relative to minimum capacity CPUs in the system and not the task CPU. Change-Id: Ia09840a5972881cad7ba7bea8fe34c45f909725e Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched/fair: Introduce scheduler boost for low latency workloadsSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Certain low latency bursty workloads require immediate use of highest capacity CPUs in HMP systems. Existing load tracking mechanisms may be unable to respond to the sudden surge in the system load within the latency requirements. Introduce the scheduler boost feature for such workloads. While boost is in effect the scheduler bypasses regular load based task placement and prefers highest capacity CPUs in the system for all non-small fair sched class tasks. Provide both a kernel and userspace API for software that may have apriori knowledge about the system workload. Change-Id: I783f585d1f8c97219e629d9c54f712318821922f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched/sysctl.h.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Move call to trace_sched_cpu_load()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | select_best_cpu() invokes trace_sched_cpu_load() for all online cpus in a loop, before it enters the loop for core selection. Moving invocation of trace_sched_cpu_load() in inner core loop is potentially more efficient. Change-Id: Iae1c58b26632edf9ec5f5da905c31356eb95c925 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: fair: Reset balance_interval before sending NOHZ kickSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | balance_interval needs to be reset for anycpu being kicked. Otherwise it can end up ignoring the kick (i.e not doing load balance for itself). Also bypass the check for existence of idle cpus in tickless state for !CONFIG_SCHED_HMP to allow for more aggressive load balance. Change-Id: I52365ee7c2997ec09bd93c4e9ae0293a954e39a8 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in nohz_kick_needed().] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Avoid active migration of small tasksSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently check the need to migrate the currently running task in scheduler_tick(). Skip that check for small tasks, as its not worth the effort! Change-Id: Ic205cc6452f42fde6be6b85c3bf06a8542a73eba Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Account for cpu's current frequency when calculating its power costSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In estimating cost of running a task on a given cpu, cost of cpu at its current frequency needs to over-ride cost at frequency demanded by task, where cur_freq exceeds required frequency of task. This is because placing a task on a cpu can only result in an increase of cpu's frequency. Change-Id: I021a3bbaf179bf1ec2c7f4556870256936797eb9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: make sched_set_window() return failure when PELT is in useSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Window-based load tracking is a pre-requisite for the scheduler to feed cpu load information to the governor. When PELT is in use, return failure when governor attempts to set window-size. This will let governor fall back to other APIs for retrieving cpu load statistics. Change-Id: I0e11188594c1a54b3b7ff55447d30bfed1a01115 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial merge conflict in include/linux/sched.h.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: debug: Print additional information in /proc/sched_debugSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Provide information in /proc/sched_debug on min_capacity, max_capacity and whether pelt or window-based task load statistics is in use. Change-Id: Ie4e9450652f4c83110dda75be3ead8aa5bb355c3 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Move around codeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move up chunk of code to be defined early. This helps a subsequent patch that needs update_min_max_capacity() Change-Id: I9403c7b4dcc74ba4ef1034327241c81df97b01ea Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial conflicts.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Update capacity of all online cpus when min_max_freq changesSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During bootup, its possible for min_max_freq to change as frequency information for additional clusters is processed. That would need to trigger recalculation of capacity/load_scale_factor for all (online) cpus, as they strongly depend on min_max_freq variable. Not doing so would imply some cpus will have their capacity/load_scale_factor computed wrongly. Change-Id: Iea5a0a517a2d71be24c2c71cdd805c0733ce37f8 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: update task statistics when CPU frequency changesSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A CPU may have its frequency changed by a different CPU. Because of this, it is not guaranteed that we will update task statistics at approximately the same time that the frequency change occurs. To guard against accruing time to a task at the wrong frequency, update the task's window-based statistics if the CPU it is running on changes frequency. Change-Id: I333c3f8aa82676bd2831797b55fd7af9c4225555 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Add new trace eventsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Add trace events for update_task_ravg(), update_history(), and set_task_cpu(). These tracepoints are useful for monitoring the per-task and per-runqueue demand statistics. Change-Id: Ibec9f945074ff31d1fc1a76ae37c40c8fea8cda9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: do not balance on exec if SCHED_HMPSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rebalancing at exec time will currently undo any beneficial placement that has been done during fork time, since select_best_cpu() will not discount the currently running task. For now just skip re-evaluating task placement at exec. Change-Id: I1e5e0fcc329b7b53c338c8c73795ebd5e85a118b Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: Use historical load for freq governor inputSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Historical load maintained per task can be used to influence cpu frequency better. For example, when a heavy demand task wakes up after prolonged sleep, we could use the historical load information to alert cpufreq governor about the need to raise cpu frequency. This patch changes CPU busy statistics to be aggregation of historical task demand. Also task's historical load (as defined by sysctl_sched_window_stats_policy) is add to cpu's busy statistics (rq->curr_runnable_sum) whenever it executes on a cpu. Change-Id: I2b66136f138b147ba19083b9b044c4feb20d9b57 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: window-stats: apply scaling to full elapsed windowsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the event that a full window (or multiple full windows) have elapsed when updating a task's window-based stats, the runtime of those windows needs to be scaled based on the CPU frequency. This is currently missing, causing full windows to be accounted as having elapsed at maximum frequency, erroneously inflating task demand. Change-Id: I356b4279d44d4f39c8aea881c04327b70ed66183 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: notify cpufreq on over/underprovisioned CPUsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After a migration occurs the source and destination CPUs may not be running at frequencies which match the new task load on those CPUs. Previously, the scheduler was notifying cpufreq anytime a task greater than a certain size migrates. This is suboptimal however since this does not take into account the CPU's current frequency and other task activity that may be present. Change-Id: I5092bda3a517e1343f97e5a455957c25ee19b549 Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: Introduce spill threshold tunables to manage overcommitmentSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the number of tasks intended for a cluster exceed the number of mostly idle CPUs in that cluster, the scheduler currently freely uses CPUs in other clusters if possible. While this is optimal for performance the power trade off can be quite significant. Introduce spill threshold tunables that govern the extent to which the scheduler should attempt to contain tasks within a cluster. Change-Id: I797e6c6b2aa0c3a376dad93758abe1d587663624 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org [joonwoop@codeaurora.org: fixed conflict in nohz_kick_needed()] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: add affinity, task load information to sched tracepointsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Knowing the affinity mask and CPU usage of a task is helpful in understanding the behavior of the system. Affinity information has been added to the enq_deq trace event, and the migration tracepoint now reports the load of the task migrated. Change-Id: I29d8a610292b4dfeeb8fe16174e9d4dc196649b7 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: add migration load change notifier for frequency guidanceSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a task moves between CPUs in two different frequency domains the cpufreq governor may wish to immediately modify the frequency of both the source and destination CPUs of the migrating task. A tunable is provided to establish what size task is considered "significant" enough to warrant notifying cpufreq. Also fix a bug that would cause load to not be accounted properly during wakeup migrations. Change-Id: Ie8f6b1cc4d43a602840dac18590b42a81327c95a Signed-off-by: Steve Muckle <smuckle@codeaurora.org> [rameezmustafa@codeaurora.org: Add double rq locking for set_task_cpu()] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched/fair: Limit MAX_PINNED_INTERVAL for more frequent load balancingSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Should the system get stuck in a state where load balancing is failing due to all tasks being pinned, deferring load balancing for up to half a second may cause further performance problems. Eventually all tasks will not be pinned and load balancing should not be deferred for a great length of time. Change-Id: I06f93b5448353b5871645b9274ce4419dc9fae0f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched/fair: Help out higher capacity CPUs when they are overcommittedSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we have a task to schedule, we currently don't consider CPUs where it will not fit even if they are idle. Instead we choose the previous CPU which is sub-optimal for performance if an idle CPU is present. This change introduces tracking of any idle CPUs irrespective of whether the task fits on them or not. If we don't have a good place to put the task, prefer the lowest power idle CPU. Change-Id: I4e8290639ad1602541a44a80ba4b2804068cac0f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched/rt: Introduce power aware scheduling for real time tasksSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Real Time task scheduling has historically been geared towards performance with a significant attempt to keep higher priority tasks on the same CPU. This is not optimal for power since the task CPU may not be the most power efficient CPU. Also task movement via select_lowest_rq() gives CPU priority the primary consideration before looking at CPU topologies to find a CPU closest to the task CPU in terms of topology. This again is not optimal for power since the closest CPU may be significantly worse for power than CPUs further away. This patch removes any bias for the task CPU. When the lowest priority CPUs in the system are found we give no consideration to the CPU topology. Instead we find the lowest power CPU within local_cpu_mask. This takes care of select_task_rq_rt() and push_task(). The pull model remains unaffected since we have no room for power optimization there. Change-Id: I4162ebe2f74be14240e62476f231f9e4a18bd9e8 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: s/__get_cpu_var/this_cpu_ptr/] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: balance power inefficient CPUs with one taskSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Normally the load balancer does not pay attention to CPUs with one task since it is not possible to subdivide that load any further to achieve better balance. With power aware scheduling however it may be desirable to relocate that one task if the CPU it is currently executing on is less power efficient than other CPUs in the system. Change-Id: Idf3f5e22b88048184323513f0052827b884526b6 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org [joonwoop@codeaurora.org: group_capacity_factor changed to group_no_capacity. group_classify is called by update_sg_lb_stats() too.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: check for power inefficient task placement in tickSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Although tasks are routed to the most power-efficient CPUs during task wakeup, a CPU-bound task will not go through this decision point. Load balancing can help if it is modified to dislodge a single task from an inefficient CPU. The situation can be further improved if during the tick, the task placement is checked to see if it is optimal. This sort of checking is already being done to ensure proper task placement in heterogneous CPU topologies, so checking for power efficient task placement fits pretty well. Change-Id: I71e56d406d314702bc26dee1438c0eeda7699027 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: do nohz load balancing in order of power efficiencySteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The nohz load balancer CPU does load balancing on behalf of all idle tickless CPUs. In the interest of power efficiency though, we should do load balancing on the most power efficient idle tickless CPU first, and then work our way towards the least power efficient idle tickless CPU. This will help load find its way to the most power efficient CPUs in the system. Since when selecting the CPU to balance next it is unknown what task load would be pulled, a frequency must be assumed in order to do a comparison of CPU power consumption. The maximum freqeuncy supported by all CPUs is used for this. Change-Id: I96c7f4300fde2c677c068dc10fc0e57f763eb9b2 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in nohz_idle_balance().] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: run idle_balance() on most power-efficient CPUSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a CPU goes idle, it checks to see whether it can pull any load from other busy CPUs. The CPU going idle may not be the most power-efficient idle CPU in the system however. This patch causes the CPU going idle to check to see whether there is a more power-efficient idle CPU within the same lowest sched domain. If there is, then it runs the load balancer on behalf of that CPU instead of itself. Since it is unknown at this point what task load would be pulled, a frequency must be assumed for this in order to do a comparison of CPU power consumption. The maximum freqeuncy supported by all CPUs is used for this. Change-Id: I5eedddc1f7d10df58ecd358f37dba563eeecf4fc Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org [joonwoop@codeaurora.org: fixed minor conflict around comment.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: add hook for platform-specific CPU power informationSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To enable power-aware scheduling, provide a hook/infrastructure for platforms to communicate CPU power requirements for each supported CPU frequency. This information is then used to estimate the cost of running a task on a given CPU. Currently, an assumption is made that the task will be running by itself on the CPU. Given the current policy tries to spread tasks as much as possible this assumption should not be too far off. Change-Id: I19f1fa760a0d43222d2880f8aec0508c468b39bb Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: return rq->capacity as power cost with sched_use_pelt=1. se.avg.runnable_avg_period is deprecated and power_cost() will be changed by subsequent change anyway./ Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: add power aware scheduling sysctlSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The sched_enable_power_aware sysctl will control whether or not scheduling decisions are influenced by the power consumption of individual CPUs. Change-Id: I312f892cf76a3fccc4ecc8aa6703908b205267f0 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Extend update_task_ravg() to accept wallclock as argumentSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will make it easier to account interrupt time on a cpu, introduced in a subsequent patch. Change-Id: I0e1fb5255c280ca374fd255e7fc19d5de9f8b045 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: add sched_get_busy, sched_set_window APIsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sched_get_busy() returns the busy time of a cpu during the most recent completed window. sched_set_window() will set window size and aligns windows across all CPUs. Change-Id: Ic53e27f43fd4600109b7b6db979e1c52c7aca103 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched.h] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: adjust RQ curr, prev sums on task migrationSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Adjust cpu's busy time in its recent and previous window upon task migration. This would enable scheduler to provide better inputs to cpufreq governor on a cpu's busy time in a given window. Change-Id: Idec2ca459382e9f46d882da3af53148412d631c6 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Add aggregated runqueue windowed statsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add counters per-cpu to track its busy time in the latest window and one previous to that. This would be needed to track accurate busy time per-cpu that accounts for migrations. Basically once a task migrates, its execution time in current window is migrated as well to new cpu. The idle task's runtime is not accounted since it should not count towards runqueue busy time. Change-Id: I4014dd686f95dbbfaa4274269bc36ed716573421 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: add prev_window counter per-taskSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently windows where tasks had no execution time are ignored. However accurate accounting of cpu busy time that factors in migration would need to know actual utilization of a task in the window previous to the latest one. This would help scheduler guide cpufreq governor on busy time per-cpu that is not subject to migration induced errors. Change-Id: I5841b1732c83e83d69002139de3bdb93333ce347 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: synchronize windows across cpusSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synchronizing windows across cpus for task load measurements simplifies cpu busy time accounting during migrations. For task migrations, its usage in current window can be carried over to its new cpu. This lets cpufreq governor see a correct picture of cpu busy time that is not affected by migrations. This patch lines up windows across cpus. One of the cpu, sync_cpu, serves as a reference for all others. During bootup sync_cpu would initialize its window_start (from its sched_clock()). Other cpus will synchronize their window_start in reference to sync_cpu. This patch assumes synchronous sched_clock() across cpus and may need some change to address architectures which do not provide such synchronized sched_clock(). Change-Id: I13381389a72f5f9f85cc2446401d493a55c78ab7 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Do not account wait timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Task load statistics are used for two purposes : cpu frequency management and placement. Task's load can't be accurately judged by its wait time. For ex: a task could have waited for 10ms and when given opportunity to run, could just execute for 1ms. Accounting for 11ms as task's demand could be over-stating its needs in this example. For now, remove wait time from task demand and instead let task load be derived from its actual exec time. This may need to become a tunable feature. Change-Id: I47e94c444c6b44c3b0810347287d50f1ee685038 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: update during migration and earlier at wakeupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During migrations accounting needs to be done in set_task_cpu() to subtract the task activity from the source CPU and add it to the destination CPU. This accounting will require that the task's window based load statistics be up to date. Unfortunately, the window-based statistics cannot always be updated in set_task_cpu() because they are already being updated in the wakeup path. We cannot update the statistics solely in the wakeup path because not all wakeups are migrations. Those non-migrating wakeups will not enter set_task_cpu(). To ensure the window-based stats are always updated for both wakeup migrations and regular migrations, they are updated earlier in the wakeup path, and also updated in set_task_cpu if the task is already runnable (this ensures it is not a wakeup migration, but a regular migration). Change-Id: Ib246028741d0be9bb38ce93679d6e6ba25b10756 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in fair.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: move definition of update_task_ravg()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | set_task_cpu() will need to call update_task_ravg(). Move up definition to make it easy. Change-Id: I95c1c9e009bd1805f28708e8d6fd3b7b2166410e Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Switch to windows based load stats by defaultSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Set window-based load stats to be the default mechanism under which tasks get classified (as big/small) and which will drive frequency demand for tasks. sched_ravg_window kernel parameter can be used to change this default setting to use PELT (per-entity load tracking) scheme instead. Change-Id: I626110daa0bb2b53172bedea829d31877255ceaa Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
generated by cgit v1.2.3 (git 2.25.1) at 2026-01-14 05:06:12 +0000