summaryrefslogtreecommitdiff
path: root/kernel/sched (follow)
Commit message (Collapse)AuthorAge
...
* | | sched: window-stats: Remove unused prev_window variableSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | Remove unused prev_window variable in 'struct ravg' Change-Id: I22ec040bae6fa5810f9f8771aa1cb873a2183746 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: disable frequency notifications by defaultSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The frequency notifications from the scheduler do not currently respect synchronous topologies. If demand on CPU 0 is driving frequency high and CPU 1 is in the same frequency domain, and demand on CPU 1 is low, frequency notifiers will be continuously sent by CPU 1 in an attempt to have its frequency lowered. Until the notifiers are fixed, disable them by default. They can still be re-enabled at runtime. Change-Id: Ic8a927af2236d8fe83b4f4a633b20a8ddcfba359 Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: fix misalignment between requested and actual windowsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When set_window_start() is first executed sched_clock() has not yet stabilized. Refresh the sched_init_jiffy and sched_clock_at_init_jiffy values until it is known that sched_clock has stabilized - this will be the case by the time a client calls the sched_set_window() API. Change-Id: Icd057707ff44c3b240e5e7e96891b23c95733daa Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched: Make RAVG_HIST_SIZE tunableOlav Haugan2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Make RAVG_HIST_SIZE available from /proc/sys/kernel/sched_ravg_hist_size to allow tuning of the size of the history that is used in computation of task demand. CRs-fixed: 706138 Change-Id: Id54c1e4b6e974a62d787070a0af1b4e8ce3b4be6 Signed-off-by: Olav Haugan <ohaugan@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in sysctl.h] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Fix possibility of "stuck" reserved flagSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | check_for_migration() could mark a thread for migration (in rq->push_task) and invoke active_load_balance_cpu_stop(). However that thread could get migrated to another cpu by the time active_load_balance_cpu_stop() runs, which could fail to clear reserved flag for a cpu and drop task_sruct reference when cpu has only one task (stopper thread running active_load_balance_cpu_stop()). This would cause a cpu to have reserved bit stuck, which prevents it from being used effectively. Fix this by having active_load_balance_cpu_stop() drop reserved bit always. Change-Id: I2464a46b4ddb52376a95518bcc95dd9768e891f9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: initialize env->flags variable to 0Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | env->flags and env->new_dst_cpu fields are not initialized in load_balance() function. As a result, load_balance() could wrongly see LBF_SOME_PINNED flag set and access (bogus) new_dst_cpu's runqueue leading to invalid memory reference. Fix this by initializing env->flags field to 0. While we are at it, fix similar issue in active_load_balance_cpu_stop() function, although there is no harm present currently in that function with uninitialized env->flags variable. Change-Id: Ied470b0abd65bf2ecfa33fa991ba554a5393f649 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: 64-bit type for curr/prev_runnable_sumSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Expand rq->curr_runnable_sum and rq->prev_runnable_sum to be 64-bit counters as otherwise they can easily overflow when a cpu has many tasks. Change-Id: I68ab2658ac6a3174ddb395888ecd6bf70ca70473 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Allow acct_wait_time to be tunedSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | Add sysctl interface to tune sched_acct_wait_time variable at runtime Change-Id: I38339cdb388a507019e429709a7c28e80b5b3585 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Account interrupt handling time as busy timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Account cycles spent by idle cpu handling interrupts (irq or softirq) towards its busy time. Change-Id: I84cc084ced67502e1cfa7037594f29ed2305b2b1 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: Account idle time as busy timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Provide a knob to consider idle time as busy time, when cpu becomes idle as a result of io_schedule() call. This will let governor parameter 'io_is_busy' to be appropriately honored. Change-Id: Id9fb4fe448e8e4909696aa8a3be5a165ad7529d3 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Account wait timeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Extend window-based task load accounting mechanism to include wait-time as part of task demand. A subsequent patch will make this feature configurable at runtime. Change-Id: I8e79337c30a19921d5c5527a79ac0133b385f8a9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: update task demand on tickSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A task can execute on a cpu for a long time without being preempted or migrated. In such case, its demand can become outdated for a long time. Prevent that from happening by updating demand of currently running task during scheduler tick. Change-Id: I321917b4590635c0a612560e3a1baf1e6921e792 CRs-Fixed: 698662 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial merge conflict in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Fix herding issueSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | check_for_migration() could run concurrently on multiple cpus, resulting in multiple tasks wanting to migrate to same cpu. This could cause cpus to be underutilized and lead to increased scheduling latencies for tasks. Fix this by serializing select_best_cpu() calls from cpus running check_for_migration() check and marking selected cpus as reserved, so that subsequent call to select_best_cpu() from check_for_migration() will skip reserved cpus. Change-Id: I73a22cacab32dee3c14267a98b700f572aa3900c Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: window-stats: print window size in /proc/sched_debugSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Printing window size in /proc/sched_debug would provide useful information to debug scheduler issues. Change-Id: Ia12ab2cb544f41a61c8a1d87bf821b85a19e09fd Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Extend ftrace event to record boost and reason codeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Add a new ftrace event to record changes to boost setting. Also extend sched_task_load() ftrace event to record boost setting and reason code passed to select_best_cpu(). This will be useful for debug purpose. Change-Id: Idac72f86d954472abe9f88a8db184343b7730287 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Avoid needless migrationSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Restrict check_for_migration() to operate on fair_sched class tasks only. Also check_for_migration() can result in a call to select_best_cpu() to look for a better cpu for currently running task on a cpu. However select_best_cpu() can end up suggesting a cpu that is not necessarily better than the cpu on which task is running currently. This will result in unnecessary migration. Prevent that from happening. Change-Id: I391cdda0d7285671d5f79aa2da12eaaa6cae42d7 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Drop active balance request upon cpu going offlineSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A cpu could mark its currently running task to be migrated to another cpu (via rq->push_task/rq->push_cpu) and could go offline before active load balance handles the request. In such case, clear the active load balance request. Change-Id: Ia3e668e34edbeb91d8559c1abb4cbffa25b1830b Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: trigger immediate migration of tasks upon boostSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently turning on boost does not immediately trigger migration of tasks from lower capacity cpus. Tasks could incur migration latency of up to one timer tick (when check_for_migration() is run). Fix this by triggering a migration check on cpus with lower capacity as soon as boost is turned on for first time. Change-Id: I244649f9cb6608862d87631325967b887b7f4b7e Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: Extend boost benefit for small and low-prio tasksSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Allow small and low-prio tasks to benefit from boost, which is expected to last for a short duration. Any task that wishes to run during that short period is allowed boost benefit. Change-Id: I02979a0c5feeba0f1256b7ee3d73f6b283fcfafa Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Handle policy change properlySrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | sched_window_stat_policy influences task demand and thus various statistics maintained per-cpu like curr_runnable_sum. Changing policy non-atomically would lead to improper accounting. For example, when task is enqueued on a cpu's runqueue, its demand that is added to rq->cumulative_runnable_avg could be based on AVG policy and when its dequeued its demand that is removed can be based on MAX, leading to erroneous accounting. This change causes policy change to be "atomic" i.e all cpu's rq->lock are held and all task's window-stats are reset before policy is changed. Change-Id: I6a3e4fb7bc299dfc5c367693b5717a1ef518c32d CRs-Fixed: 687409 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched/sysctl.h. Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: window-stats: Reset all window statsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, few of the window statistics for tasks are not reset when window size is changing. Fix this to completely reset all window statistics for tasks and cpus. Move the reset code to a function, which can be reused by a subsequent patch that resets same statistics upon policy change. Change-Id: Ic626260245b89007c4d70b9a07ebd577e217f283 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Additional error checking in sched_set_window()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Check for invalid window size passed as argument to sched_set_window() Also move up local_irq_disable() call to avoid thread from being preempted during calculation of window_start and its comparison against sched_clock(). Use right macro to evluate whether window_start argument is ahead in time or not. Change-Id: Idc0d3ab17ede08471ae63b72a2d55e7f84868fd6 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Fix incorrect calculation of partial_demandSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | When using MAX_POLICY, partial_demand is calculated incorrectly as 0. Fix this by picking maximum of previous 4 windows and most recent sample. Change-Id: I27850a510746a63b5382c84761920fc021b876c5 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: window-stats: Fix potential wrong use of rqSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 'rq' reference to a cpu where a waking task last ran can be potentially incorrect leading to incorrect accounting. This happens when task_cpu() changes between points A & B in try_to_wake_up() listed below: try_to_wake_up() { cpu = src_cpu = task_cpu(p); rq = cpu_rq(src_cpu); -> Point A .. while (p->on_cpu) cpu_relax(); smp_rmb(); raw_spin_lock(&rq->lock); -> Point B Fix this by initializing 'rq' variable after task has slept (its on_cpu field becomes 0). Also avoid adding task demand to its old cpu runqueue (prev_runnable_sum) in case it's gone offline. Change-Id: I9e5d3beeca01796d944137b5416805b983a6e06e Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: set initial task load to just above a small taskSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To maximize power savings, set the intial load of newly created tasks to just above a small task. Setting it below the small task threshold would cause new tasks to be packed which is very likely too aggressive. Change-Id: Idace26cc0252e31a5472c73534d2f5277a1e3fa4 Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
* | | sched/fair: Check whether any CPUs are availableOlav Haugan2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a possibility that there are no allowed CPUs online when we try to select the best cpu for a small task. Add a check to ensure we don't continue if there are no CPUs available. CRs-fixed: 692505 Change-Id: Iff955fb0d0b07e758a893539f7bc8ea8aa09d9c4 Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
* | | sched: enable hmp, power aware scheduling for targets with > 4 CPUsSteve Muckle2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Enabling and disabling hmp/power-aware scheduling is meant to be done via kernel command line options. Until that is fully supported however, take advantage of the fact that current targets with more than 4 CPUs will need these features. Change-Id: I4916805881d58eeb54747e4b972816ffc96caae7 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: remove sysctl control for HMP and power-aware task placementSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is no real need to control HMP and power-aware task placement at runtime after kernel has booted. Boot-time control should be sufficient. Not allowing for runtime (sysctl) support simplifies the code quite a bit. Also rename sysctl_sched_enable_hmp_task_placement to be shorter. Change-Id: I60cae51a173c6f73b79cbf90c50ddd41a27604aa Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict. p->nr_cpus_allowed == 1 has moved to core.c Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: support legacy mode betterSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It should be possible to bypass all HMP scheduler changes at runtime by setting sysctl_sched_enable_hmp_task_placement and sysctl_sched_enable_power_aware to 0. Fix various code paths to honor this requirement. Change-Id: I74254e68582b3f9f1b84661baf7dae14f981c025 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed conflict in rt.c, p->nr_cpus_allowed == 1 is now moved in core.c] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: code cleanupSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Avoid the long if() block of code in set_task_cpu(). Move that code to its own function Change-Id: Ia80a99867ff9c23a614635e366777759abaccee4 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Add BUG_ON when task_cpu() is incorrectSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It would be fatal if task_cpu() information for a task does not accurately represent the cpu on which its running. All sorts of wierd issues can arise if that were to happen! Add a BUG_ON() in context switch to detect such cases. Change-Id: I4eb2c96c850e2247e22f773bbb6eedb8ccafa49c Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: avoid active migration of tasks not in TASK_RUNNING stateSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Avoid wasting effort in migrating tasks that are about to sleep. Change-Id: Icf9520b1c8fa48d3e071cb9fa1c5526b3b36ff16 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: fix up task load during migrationSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix the hack to set task's on_rq to 0 during task migration. Task's load is temporarily added back to its runqueue so that update_task_ravg() can fixup task's load when its demand is changing. Task's load is removed immediately afterwards. Temporarily setting p->on_rq to 0 introduces a race condition with try_to_wake_up(). Another task (task A) may be attempting to wake up the migrating task (task B). As long as task A sees task B's p->on_rq as 1, the wake up will not continue. Changing p->on_rq to 0, then back to 1, allows task A to continue "waking" task B, at which point we have both try_to_wake_up and the migration code attempting to set the cpu of task B at the same time. CRs-Fixed: 695071 Change-Id: I525745f144da4ffeba1d539890b4d46720ec3ef1 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: avoid pushing tasks to an offline CPUPrasad Sodagudi2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently active_load_balance_cpu_stop is run by cpu stopper and it pushes running tasks off the busiest CPU onto idle target CPU. But there is no check to see whether target cpu is offline or not before pushing the tasks. With the introduction of active migration in the scheduler tick path (see check_for_migration()) there have been instances of attempts to migrate tasks to offline CPUs. Add a check as to whether the target cpu is online or not to prevent scheduling on offline CPUs. Change-Id: Ib8ac7f8aeabd3ca7365f3eae977075952dab4f21 Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org> [rameezmustafa@codeaurora.org]: Port to msm-3.18] Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
* | | sched: Add a per rq max_possible_capacity for use in power calculationsSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the absence of a power driver providing real power values, the scheduler currently defaults to using capacity of a CPU as a measure of power. This, however, is not a good measure since the capacity of a CPU can change due to thermal conditions and/or other hardware restrictions. These frequency restrictions have no effect on the power efficiency of those CPUs. Introduce max possible capacity of a CPU to track an absolute measure of capacity which translates into a good absolute measure of power efficiency. Max possible capacity takes the max possible frequency of CPUs into account instead of max frequency. Change-Id: Ia970b853e43a90eb8cc6fd990b5c47fca7e50db8 Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Disable interrupts when holding the rq lock in sched_get_busy()Syed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Interrupts can end up waking processes on the same cpu as the one for which sched_get_busy() is called. Since sched_get_busy() takes the rq lock this can result in a deadlock as the same rq lock is required to enqueue the waking up task. Fix the deadlock by disabling interrupts when taking the rq lock. Change-Id: I46e14a14789c2fb0ead42363fbaaa0a303a5818f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Make wallclock more accurateSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | update_task_ravg() in context switch uses wallclock that is updated before running put_prev_task() and pick_next_task(), both of which can take some time. Its better to update wallclock after those routines, leading to more accurate accounting. Change-Id: I882b1f0e8eddd2cc17d42ca2ab8f7a2841b8d89a Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Make task and CPU load calculations safe from truncationSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Load calculations have been modified to accept and return 64 bit values. Fix up all the places where we make such calculations to store the result in 64 bit variables. This is necessary to avoid issues caused by truncation of values. While at it update scale_task_load() to scale_load_to_cpu(). This is because the API is used to scale load of both individual tasks as well as the cumulative load of CPUs. In this sense the name was a misnomer. Also clean up power_cost() to use max_task_load(). Change-Id: I51e683e1592a5ea3c4e4b2b06d7a7339a49cce9c Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed conflict in power_cost(). power_cost() now supports sched_use_pelt=1 back again. Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched/fair: Introduce C-state aware task placement for small tasksSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Small tasks execute for small durations. This means that the power cost of taking CPUs out of a low power mode outweigh any performance advantage of using an idle core or power advantage of using the most power efficient CPU. Introduce C-state aware task placement for small tasks. This requires a two pass approach where we first determine the most power effecient CPU and establish a band of CPUs offering a similar power cost for the task. The order of preference then is as follows: 1) Any mostly idle CPU in active C-state in the same power band. 2) A CPU with the shallowest C-state in the same power band. 3) A CPU with the least load in the same power band. 4) Lowest power CPU in a higher power band. The patch also modifies the definition of a small task. Small tasks are now determined relative to minimum capacity CPUs in the system and not the task CPU. Change-Id: Ia09840a5972881cad7ba7bea8fe34c45f909725e Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched/fair: Introduce scheduler boost for low latency workloadsSyed Rameez Mustafa2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Certain low latency bursty workloads require immediate use of highest capacity CPUs in HMP systems. Existing load tracking mechanisms may be unable to respond to the sudden surge in the system load within the latency requirements. Introduce the scheduler boost feature for such workloads. While boost is in effect the scheduler bypasses regular load based task placement and prefers highest capacity CPUs in the system for all non-small fair sched class tasks. Provide both a kernel and userspace API for software that may have apriori knowledge about the system workload. Change-Id: I783f585d1f8c97219e629d9c54f712318821922f Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched/sysctl.h.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Move call to trace_sched_cpu_load()Srivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | select_best_cpu() invokes trace_sched_cpu_load() for all online cpus in a loop, before it enters the loop for core selection. Moving invocation of trace_sched_cpu_load() in inner core loop is potentially more efficient. Change-Id: Iae1c58b26632edf9ec5f5da905c31356eb95c925 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: fair: Reset balance_interval before sending NOHZ kickSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | balance_interval needs to be reset for anycpu being kicked. Otherwise it can end up ignoring the kick (i.e not doing load balance for itself). Also bypass the check for existence of idle cpus in tickless state for !CONFIG_SCHED_HMP to allow for more aggressive load balance. Change-Id: I52365ee7c2997ec09bd93c4e9ae0293a954e39a8 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed minor conflict in nohz_kick_needed().] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Avoid active migration of small tasksSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We currently check the need to migrate the currently running task in scheduler_tick(). Skip that check for small tasks, as its not worth the effort! Change-Id: Ic205cc6452f42fde6be6b85c3bf06a8542a73eba Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Account for cpu's current frequency when calculating its power costSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In estimating cost of running a task on a given cpu, cost of cpu at its current frequency needs to over-ride cost at frequency demanded by task, where cur_freq exceeds required frequency of task. This is because placing a task on a cpu can only result in an increase of cpu's frequency. Change-Id: I021a3bbaf179bf1ec2c7f4556870256936797eb9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: make sched_set_window() return failure when PELT is in useSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Window-based load tracking is a pre-requisite for the scheduler to feed cpu load information to the governor. When PELT is in use, return failure when governor attempts to set window-size. This will let governor fall back to other APIs for retrieving cpu load statistics. Change-Id: I0e11188594c1a54b3b7ff55447d30bfed1a01115 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial merge conflict in include/linux/sched.h.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: debug: Print additional information in /proc/sched_debugSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | Provide information in /proc/sched_debug on min_capacity, max_capacity and whether pelt or window-based task load statistics is in use. Change-Id: Ie4e9450652f4c83110dda75be3ead8aa5bb355c3 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: Move around codeSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move up chunk of code to be defined early. This helps a subsequent patch that needs update_min_max_capacity() Change-Id: I9403c7b4dcc74ba4ef1034327241c81df97b01ea Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org> [joonwoop@codeaurora.org: fixed trivial conflicts.] Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
* | | sched: Update capacity of all online cpus when min_max_freq changesSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | During bootup, its possible for min_max_freq to change as frequency information for additional clusters is processed. That would need to trigger recalculation of capacity/load_scale_factor for all (online) cpus, as they strongly depend on min_max_freq variable. Not doing so would imply some cpus will have their capacity/load_scale_factor computed wrongly. Change-Id: Iea5a0a517a2d71be24c2c71cdd805c0733ce37f8 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
* | | sched: update task statistics when CPU frequency changesSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A CPU may have its frequency changed by a different CPU. Because of this, it is not guaranteed that we will update task statistics at approximately the same time that the frequency change occurs. To guard against accruing time to a task at the wrong frequency, update the task's window-based statistics if the CPU it is running on changes frequency. Change-Id: I333c3f8aa82676bd2831797b55fd7af9c4225555 Signed-off-by: Steve Muckle <smuckle@codeaurora.org> Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
* | | sched: Add new trace eventsSrivatsa Vaddagiri2016-03-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | Add trace events for update_task_ravg(), update_history(), and set_task_cpu(). These tracepoints are useful for monitoring the per-task and per-runqueue demand statistics. Change-Id: Ibec9f945074ff31d1fc1a76ae37c40c8fea8cda9 Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
generated by cgit v1.2.3 (git 2.25.1) at 2026-04-18 18:22:14 +0000